Forums : Technical Issue Help
Dear Open Hub Users,
We’re excited to announce that we will be moving the Open Hub Forum to
https://community.synopsys.com/s/black-duck-open-hub.
Beginning immediately, users can head over,
register,
get technical help and discuss issue pertinent to the Open Hub. Registered users can also subscribe to Open Hub announcements here.
On May 1, 2020, we will be freezing https://www.openhub.net/forums and users will not be able to create new discussions. If you have any questions and concerns, please email us at
[email protected]
NetBSD repository download failed
The repository downloads for NetBSD seem to keep failing. Originally it was pointed at anoncvs3, which we suspected may have not held the full tree, so we switched it to anoncvs. However, the downloads appear to have failed again. Could someone please look at them?
Hi garbled,
I restarted these downloads. We didn't get very far on the first try -- only able to download a few revisions on each of them. So I'm not overly hopeful that we'll make much more progress this time. Perhaps anoncvs is tuned to drop connections with high load a little sooner than anoncvs3 was?
Unfortunately, I don't think anoncvs3 was working either. The information was very sporadic, and seemed to only cover parts of the tree, and a large number of commits were missing. I wonder if there would be another way we could get you the data you need. Is an rsync of the anoncvs repository a viable alternative for you?
It's technically possible but expensive. I've spent about 20-30 hours so far trying to do something similar for OpenBSD (using cvsync) and it's still quite a way from success. When I get that finished (unfortunately no ETA on that), I might be able to circle back to NetBSD and do something similar. In preparation for that, do you know if there are any CVSYNC servers for NetBSD and their urls?
Thanks!
No, we don't have one of those running that I know of. We have anoncvs, cvsup, and rsync available. See http://www.netbsd.org/mirrors/ for the urls for those.
As for anoncvs.. is it just dropping your connection? Do you have any errors or the like I could pass on to our admins?
We don't have cvsync on anoncvs.netbsd.org, and I'm not aware of a mirror offering it either.
People wanting the entire repository usually use rsync, like garbled mentioned already.
Please consider the size of the repository (du -sk)
785269 htdocs/
28621 othersrc/
2920266 src/
958628 xsrc/
Would starting you out with tarballs (for ftp) help?
@garbled: it seems to just be dropping the connection. We're able to pull a few revisions and then we get:
connect to anoncvs.netbsd.org:2401 failed: Network is unreachable
@S.P.Zeidler: Am I correct to assume that rsync or a tarball has a similar purpose as CVSYNC: provide a full copy of the repository, including all changes, such that you can now operate as a mirror? If so, the value to Ohloh would be that we could then pull from that (internal) mirror to start the project?
Andy: the purpose of the tarball would be to get the bulk of the data to you with a protocol that does reget.
I'd expect you to be continuing with rsync after; that would only transfer subsequent changes (and a bit of meta-info like checksums). Our anoncvs mirrors also use rsync (and sometimes also offer rsync).
Give me a nudge when you have time for the ftp job, I'll put up fresh tarballs for you then.
network is unreachable is different. This seems more like a connectivity issue than anoncvs dumping the connection on you. Can you do the traditional traceroute and try firing off just one scan at a time?
I already tried a traceroute in the opposite direction (assuming www.ohloh.net is the target) and didn't see anything obvious:
- exit.sf-nbsd.sfo2.isc.org (204.152.190.1) 0.342 ms 0.396 ms 0.243 ms
- int-0-1-0-606.r3.sfo2.isc.org (149.20.65.4) 1.093 ms 0.914 ms 0.976 ms
- int-0-1-0.r4.sfo2.isc.org (149.20.65.22) 20.845 ms 19.330 ms 19.406 ms
- int-0-3-0.r5.sfo2.isc.org (149.20.65.26) 4.019 ms 22.739 ms 20.108 ms
- int-0-1-0.r1.pao1.isc.org (149.20.65.21) 1.531 ms 1.470 ms 1.561 ms
- ge-1-11.r03.plalca01.us.bb.gin.ntt.net (140.174.21.193) 1.680 ms 1.791 ms 1.846 ms
- xe-2-1-0.r21.plalca01.us.bb.gin.ntt.net (129.250.4.245) 1.971 ms 1.840 ms 1.851 ms
- ae-1.r20.snjsca04.us.bb.gin.ntt.net (129.250.5.33) 2.557 ms 2.885 ms 2.723 ms
- p64-0-0-0.r20.sttlwa01.us.bb.gin.ntt.net (129.250.4.22) 21.873 ms 21.924 ms 21.888 ms
- po-2.r00.sttlwa01.us.bb.gin.ntt.net (129.250.2.205) 21.865 ms 21.891 ms 23.200 ms
- ge-0.isomedia.r00.sttlwa01.us.ce.gin.ntt.net (204.2.144.10) 21.571 ms 21.561 ms 21.302 ms
- core-01-ge-0-2.sttl.isomedia.com (207.115.64.163) 21.424 ms 21.145 ms 21.289 ms
- gate-02-pos-4-0.rdmd.isomedia.com (207.115.66.73) 23.034 ms 22.997 ms 22.337 ms
- red-ptp.isomedia.com (207.115.64.101) 23.616 ms 23.294 ms 23.798 ms
- 207.115.86.98 (207.115.86.98) 22.744 ms 24.004 ms 23.953 ms
(I don't expect there to be a problem with ISC or NTT. I don't know isomedia.com and wether they have a 'suspicious network activity' filter going, but I'd assume the ohloh.net people would have noticed previously if that was the case.)
Also, the data needs to get to ohloh.net somehow, and ftp reget does have its advantages in that it'll let you proceed (and before you're old and gray, too :-P )
allbsd.org has cvsync access if that is helpful.
I've now thrown a different approach at the problem, which seems to work:
pkgsrc is now enlisted as 58 repositories, each category on its own, and I'm doing the same to the directories under src (where we will end up with 30 repositories for src unless I need to split src/sys and src/gnu which will make it worse).
I notice that restarting the “downloads” takes almost as long as starting them in the first place, and places a heavy burden on the server (if only one project is synching, two cvs processes at about 50% CPU each).
Maybe you should really use direct comma-v file download (using rsyncd or anonrsync-over-ssh) and parsing (as per rcsfile(5) or using the GNU RCS tools like rlog, rcsdiff, co) instead if the project offers such thing.
That said, the MirOS downloads are slowly progressing. Let’s see where this ends. MirPorts, with the bigger amount of (smaller individual) files already finished, so this is not necessarily the problem.
