Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go »
No member browsing this thread
Thread Status: Active
Total posts in this thread: 100
Posts: 100   Pages: 10   [ Previous Page | 1 2 3 4 5 6 7 8 9 10 | Next Page ]
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 18323 times and has 99 replies Next Thread
KerSamson
Master Cruncher
Switzerland
Joined: Jan 29, 2007
Post Count: 1677
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: WU Characteristics

Hello Wolfman1360,
for monitoring purpose, you could use BoincTask.
Regarding the poor MIP1 performances, I am really disappointed having to notice that nobody really take care of the concerning observations.
I feel that bad science is disrespectful to the contributors. It is not because plenty of GigaFlops are available at WCG, that the scientists should not take care of computational efficiency.
I spoke with colleagues at the university a couple of days ago, and they came exactly to a similar conclusion that I did over 4 weeks ago. In the mean time, nobody really took notice of this situation, only mentioning that hyperthreading could have a negative performance impact. Finally, it is a useless advice since other sciences run well and efficient with HT and because we can show that even without HT the performances are very bad.
Cheers,
Yves
----------------------------------------
[Oct 14, 2017 7:26:49 AM]   Link   Report threatening or abusive post: please login first  Go to top 
alged
Master Cruncher
FRANCE
Joined: Jun 12, 2009
Post Count: 2360
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: WU Characteristics

Well i wonder if with my little means i shud swap to SCC instead of trying just to get a badge on MIP (in fact just to show i am participing)
is it useful or not...
----------------------------------------

[Oct 14, 2017 10:27:05 AM]   Link   Report threatening or abusive post: please login first  Go to top 
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 261
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: WU Characteristics

I'm not seeing those sort of differences between MIP and other projects...

Project: Points: Runtime: ((y:d:h:m:s))
ZIK1 19,922,130 20:002:18:51:15
MIP1 1,769,834 1:168:20:38:09
SCC1 5,856,483 5:044:19:22:39
OET1 9,784,941 10:005:08:53:47

Same mix of machines for all projects - mostly Intel with hyperthreading enabled...
Majority of MIP run as the only project on the machines... (that was the only way to get enought WUs).

Edit: Forgot to mention - checked the efficiency for MIP1 and as of this moment averaging between 0.967 and 0.992 - over 24 cores
----------------------------------------
----------------------------------------
[Edit 3 times, last edit by TonyEllis at Oct 14, 2017 11:54:47 AM]
[Oct 14, 2017 11:47:39 AM]   Link   Report threatening or abusive post: please login first  Go to top 
KerSamson
Master Cruncher
Switzerland
Joined: Jan 29, 2007
Post Count: 1677
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: WU Characteristics

Hi TonyEllis,
are your machines running Windows or Linux?
On my side, in case of MIP1 only computation, the both platforms are impacted, nonetheless Linux seems to be more negatively impacted than Windows.
If you monitor the used RAM, you will notice that MIP1 is really demanding, close to 8 GB RAM, incl. Windows 7 Pro x64, for 8 threads.
On Linux hosts, I did noticed that the CPU temperature drops down even if the CPU load is still reported to be 100%.
MIP1 has definitely a memory management problem.
Maybe it could be corrected with other compiling options of the science, maybe a better (more efficient) software design should be considered, avoiding cache faults as far as possible.
Cheers,
Yves
----------------------------------------
[Oct 14, 2017 4:06:56 PM]   Link   Report threatening or abusive post: please login first  Go to top 
wolfman1360
Senior Cruncher
Canada
Joined: Jan 17, 2016
Post Count: 176
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: WU Characteristics

Seems that intel processors are much quicker at computing on WCG in general.
An i7 6500U at 2.8 ghz is very nearly beating my fx8350.
Meanwhile an i5 2300 from 2012 is beating them both.
I guess passmark benchmarks aren't everything ;).
----------------------------------------
Crunching for the betterment of human kind and the canines who will always be our best friends.
AWOU!
[Oct 14, 2017 7:22:25 PM]   Link   Report threatening or abusive post: please login first  Go to top 
ThreadRipper
Veteran Cruncher
Sweden
Joined: Apr 26, 2007
Post Count: 1322
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: WU Characteristics

But FX8350 is one generation old from AMD now (Bulldozer architecture was a notorious low-IPC:er). Ryzen is much, much better (even if it is not up to par on IPC when compared to latest Intel chips). But, since WCG crunching is an application that scales Very Well with increased core/thread count capabilities of a CPU, you will probably get more out of an AMD today in terms of WCG performance/$.
----------------------------------------

Join The International Team: https://www.worldcommunitygrid.org/team/viewTeamInfo.do?teamId=CK9RP1BKX1

AMD TR2990WX @ PBO, 64GB Quad 3200MHz 14-17-17-17-1T, RX6900XT @ Stock
AMD 3800X @ PBO
AMD 2700X @ 4GHz
[Oct 14, 2017 10:03:02 PM]   Link   Report threatening or abusive post: please login first  Go to top 
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 261
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: WU Characteristics

KerSamson - the OS mix varies slightly depending on which machines are in use...
As of this morning...

Qty #Cores CPU OS Memeory
x2 4 ea Intel Linux - Redhat Enterprise derivative version 6.9 - firewalls 2G/4G
x1 8 Intel Linux - Redhat Enterprise derivative version 6.9 - main server 8G
x1 4 Intel Redhat Fedora - my Workstation 8G
x1 4 AMD Windows 8.1 - my laptop 8G
x1 2 AMD Linux - Redhat Enterprise derivative version 7.3 - backup server 8G
26 cores

Notes...
All Intel machines with hyper-threading enabled
AMD hyper-threading not available
Windows 8.1 (and 10) have much better memory management than Windows 7
If anything would say WIP increases CPU temperature slightly
Agree WIP uses more memory as reported here https://www.worldcommunitygrid.org/forums/wcg...ead,40249_offset,0#549593 - however, that doesn't prove the project has a memory management problem.
Actually, looking in the 'project' directory for boinc
scc1 17 files using 7.6M
mip1 266 files using 495M

All machine details and performance data is available at http://www.sraellis.tk
EDIT: added memory detail for each machine & correction...
----------------------------------------
----------------------------------------
[Edit 4 times, last edit by TonyEllis at Oct 18, 2017 1:59:18 AM]
[Oct 14, 2017 10:07:47 PM]   Link   Report threatening or abusive post: please login first  Go to top 
wolfman1360
Senior Cruncher
Canada
Joined: Jan 17, 2016
Post Count: 176
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: WU Characteristics

I'm very much looking forward to ryzen notebook CPU's. Does anyone have info on them/release date?
I think exciting times are ahead.
----------------------------------------
Crunching for the betterment of human kind and the canines who will always be our best friends.
AWOU!
[Oct 14, 2017 11:24:36 PM]   Link   Report threatening or abusive post: please login first  Go to top 
KerSamson
Master Cruncher
Switzerland
Joined: Jan 29, 2007
Post Count: 1677
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: WU Characteristics

TonyEllis,
at first, I thank you for sharing your impressive and comprehensive monitoring tools. It is on my Christmas wish list since many years, just missing available time to implement it.
I notice that the most of your hosts - excepted the i7 3770 - have only few cores, in contrary to my own hosts, with at least 4 cores for the Athlon II, 6 cores for the Phenom II, and 4 cores + HT for the both i7.
I will be again away over during the coming week, however, I would like to try to verify if MIP1 causes a significant performance decreasing based on the number of involved core; i.e. maybe the MIP1 performance impact - comparing to other sciences - could be for example -7% for one thread, -14% for two concurrent threads, -20% for three concurrent threads, -30% for four concurrent threads, etc.
Cheers,
Yves
----------------------------------------
[Oct 15, 2017 8:12:47 AM]   Link   Report threatening or abusive post: please login first  Go to top 
TonyEllis
Senior Cruncher
Australia
Joined: Jul 9, 2008
Post Count: 261
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: WU Characteristics

In an attempt to see difference between SCC1 and MIP1, since have two virtually identical machines, ran SCC1 on one and MIP1 on the other. They each have 2G memory, same motherboard, same processor with 4 CPUs (2 Intel cores hyper-threaded). An application known to run continuously was added to see what normal variations could be expected...
The boinc files for each project and the space taken for each project under "projects'
SCC1 17 files 7.6 MB
MIP1 140 files 334 MB
First ,lets see the difference in page faults and memory use over a 30 minute (1800 secs) period.

Command in this format "pidstat -p pid1,pid2,pid3,pid4 -r -t 1800 1
-r Report page faults and memory utilization. (truncated from 'man' pages)
... snipped
minft/s Total number of minor faults the task has made per second,
those which have not required loading a memory page from disk.
majflt/s Total number of major faults the task has made per second,
those which have required loading a memory page from disk.
VSZ Virtual Size: The virtual memory usage of entire task in kilobytes.
RSS Resident Set Size: The non-swapped physical memory used by the task in kilobytes.
... snipped

Average: TGID TID minflt/s majflt/s VSZ RSS %MEM Command
Average: 4427 - 1.27 0.00 284780 92280 4.53 wcgrid_mip1_ros
Average: - 4427 1.20 0.00 284780 92280 4.53 |__wcgrid_mip1_ros
Average: - 4428 0.07 0.00 284780 92280 4.53 |__wcgrid_mip1_ros
Average: 4376 - 0.58 0.00 286016 93604 4.59 wcgrid_mip1_ros
Average: - 4376 0.51 0.00 286016 93604 4.59 |__wcgrid_mip1_ros
Average: - 4385 0.07 0.00 286016 93604 4.59 |__wcgrid_mip1_ros
Average: 4365 - 20.90 0.00 405988 17660 0.87 boinc_client
Average: - 4365 20.86 0.00 405988 17660 0.87 |__boinc_client
Average: 603 - 10.35 0.00 99792 1756 0.09 gkrellmd
Average: - 603 7.75 0.00 99792 1756 0.09 |__gkrellmd

Average: TGID TID minflt/s majflt/s VSZ RSS %MEM Command
Average: 17305 - 0.18 0.00 127864 51200 2.50 wcgrid_scc1_vin
Average: - 17305 0.00 0.00 127864 51200 2.50 |__wcgrid_scc1_vin
Average: - 17310 0.06 0.00 127864 51200 2.50 |__wcgrid_scc1_vin
Average: - 17350 0.12 0.00 127864 51200 2.50 |__wcgrid_scc1_vin
Average: 15302 - 0.12 0.00 125376 48708 2.38 wcgrid_scc1_vin
Average: - 15302 0.00 0.00 125376 48708 2.38 |__wcgrid_scc1_vin
Average: - 15303 0.06 0.00 125376 48708 2.38 |__wcgrid_scc1_vin
Average: - 15322 0.06 0.00 125376 48708 2.38 |__wcgrid_scc1_vin
Average: 6788 - 24.01 0.00 185164 15740 0.77 boinc_client
Average: - 6788 24.01 0.00 185164 15740 0.77 |__boinc_client
Average: 3687 - 9.34 0.00 98984 1860 0.09 gkrellmd

Not a massive memory diference between SCC1 and MIP1 - roughly 2x - no substantial difference in page faults
OK - lets; look at disk IO
Commnd in this format "pidstat -p pid1,pid2,pid3,pid4 -d 1800 1
-d Report I/O statistics (kernels 2.6.20 and later only). The following values are displayed:
..snipped
kB_rd/s Number of kilobytes the task has caused to be read from disk per second.
kB_wr/s Number of kilobytes the task has caused, or shall cause to be written to disk per second.
kB_ccwr/s Number of kilobytes whose writing to disk has been cancelled by the task. This may occur when the task truncates some dirty pagecache. In this case, some IO which another task has been accounted for will not be happening.
...snipped

Average: TGID TID kB_rd/s kB_wr/s kB_ccwr/s Command
Average: 4427 - 115.21 0.47 0.00 wcgrid_mip1_ros
Average: - 4427 105.04 0.12 0.00 |__wcgrid_mip1_ros
Average: - 4428 10.18 0.36 0.00 |__wcgrid_mip1_ros
Average: 4376 - 290.83 0.34 0.00 wcgrid_mip1_ros
Average: - 4376 271.76 0.00 0.00 |__wcgrid_mip1_ros
Average: - 4385 19.07 0.34 0.00 |__wcgrid_mip1_ros
Average: 4365 - 28.41 3.83 0.00 boinc_client
Average: - 4365 27.64 3.65 0.00 |__boinc_client
Average: 603 - 3.77 0.00 0.00 gkrellmd
Average: - 603 2.30 0.00 0.00 |__gkrellmd

Average: TGID TID kB_rd/s kB_wr/s kB_ccwr/s Command
Average: 17305 - 0.00 0.55 0.06 wcgrid_scc1_vin
Average: - 17305 0.00 0.00 0.00 |__wcgrid_scc1_vin
Average: - 17310 0.00 0.17 0.00 |__wcgrid_scc1_vin
Average: - 17350 0.00 0.38 0.06 |__wcgrid_scc1_vin
Average: 15302 - 0.00 0.49 0.06 wcgrid_scc1_vin
Average: - 15302 0.00 0.00 0.00 |__wcgrid_scc1_vin
Average: - 15303 0.00 0.16 0.00 |__wcgrid_scc1_vin
Average: - 15322 0.00 0.33 0.06 |__wcgrid_scc1_vin
Average: 6788 - 0.00 3.13 0.72 boinc_client
Average: - 6788 0.00 3.13 0.72 |__boinc_client
Average: 3687 - 0.00 0.00 0.00 gkrellmd
Average: - 3687 0.00 0.00 0.00 |__gkrellmd

Well that is a difference! - looks like a really fast disk system helps MIP1.
If anyone has other ideas - will see what can be measured... sorry for the length...
EDIT: Sorry about the loss of formatting - it was correct in the editor copy-and-pasted from..
Update: trying the code tag suggestion below...
----------------------------------------
----------------------------------------
[Edit 3 times, last edit by TonyEllis at Oct 18, 2017 1:17:08 PM]
[Oct 18, 2017 3:32:31 AM]   Link   Report threatening or abusive post: please login first  Go to top 
Posts: 100   Pages: 10   [ Previous Page | 1 2 3 4 5 6 7 8 9 10 | Next Page ]
[ Jump to Last Post ]
Post new Thread