Transferring files between two people can be an extraordinarily painful process. Email seems perfectly reasonable, but is subject to message file attachment limits for both parties. Using file sharing technology like FTP could require setting up a dropbox or an account. If you want to use your own computer as a server (running AppleShare, for instance), you need a reachable IP address, not always a readily available commodity.
Two new services, Civil Netizen and Pando, hope to make file transfer easier by employing peer-to-peer (P2P) techniques to push data securely over the Internet. Both are in beta testing; the former turns a computer, briefly, into a peer-to-peer surfer for individual connections, and the latter acts as a time-limited central repository with distributed P2P properties for files up to 1 GB in size (during testing).
Both services are currently free, and their developers currently have no plans to charge in the future, although Pando may use some sponsorship advertising to fund bandwidth, and plans to license its technology to other firms. Civil Netizen is an open-source project with no fees.
Civil Netizen — The phrase “peer-to-peer file sharing” has taken a beating. In common use, it almost always implies illegal transfer of copyrighted materials or the implication of that act. But people using ordinary computers to transfer files are peers, and Civil Engines Research recognizes that in the choice of the name Civil Netizen for its project.
When you install Civil Netizen, you’re not setting up a server. Rather, you’re creating a specific P2P engine that is active only at certain times and in response to the right queries.
Civil Netizen lets you take one or more files or folders and create a “parcel.” The parcel has an associated pickup slip, which is a sequence of obscured data. Instead of transferring the parcel directly, the program lets you use your default email application to send just the pickup slip (you can also copy the pickup slip information to the clipboard for pasting into a program like iChat, or you can save it to a file on your Desktop).
As long as the parcel remains available within Civil Netizen (which must be running), that pickup slip enables any recipient to retrieve the file; the retrieval is logged for reference. Civil Netizen doesn’t allow generic file retrieval – other Civil Netizen users can retrieve only parcels, and only those for which they have the associated pickup slips.
Whenever a recipient attempts to retrieve a parcel by loading the pickup slip details you sent them into their copy of Civil Netizen, their software creates a connection to your computer, retrieves the parcel, and stores it locally. Civil Engines Research uses a centralized storage system for the pickup slips, but that’s the only portion of the connection that’s stored in a non-P2P fashion. You can send the pickup slip to multiple recipients, and as long as you keep the parcel available within your copy of Civil Netizen, others can download it. However, there’s no confirmation that someone with the pickup slip is a legitimate recipient.
Civil Netizen uses a fairly robust method of encrypting data in transit, employing 128-bit AES (Advanced Encryption System) session keys, which are considered quite strong. The keys are negotiated using a Diffie-Hellman key exchange, which prevents interception. However, the developers don’t use a validation step that confirms there’s no man in the middle intercepting both sides of a conversation.
However unlikely interception is for most users, the lack of a validation step prevents Diffie-Hellman from being considered reliable. In correspondence with one of the developers, he said the company expects to offer user registration that would then allow an out-of-band method to provide necessary validation. I call that the “evil dictator” problem, in that without validation, you’re well protected, but not against those intent upon intercepting traffic at a governmental level. Validation wouldn’t prevent knowledge of parties transferring data, but would – by today’s standards – provide extremely high security for the contents of packages.
Civil Netizen is at beta 4 for Mac and Windows, and the company plans a Linux client. As an open-source project, clients for other platforms or other versions of the client for Mac or Windows could appear.
Pando — Pando, from Pando Networks, takes an entirely different approach with regard to where files are stored, using what initially appears to be a hub-and-spoke system and a centralized repository, but turns out to have a P2P twist.
Once you’ve downloaded the Mac software, you can create a new package composed of one or more files or folders, and then enter recipients’ email addresses. The Pando client packages your files, uploads them to Pando Networks’s servers, and notifies recipients via email. Recipients then open the attached .pando file in their copy of the Pando application to download the packaged files.
Here’s where Pando’s approach gets interesting. Rather than simply being a file server, Pando uses P2P technology much like BitTorrent to speed the download by having the recipient’s Pando client pull data from the Pando servers (which they call supernodes); from your computer, which acts as a P2P node for the file; and from other recipients of the file who have already downloaded it (or pieces of it) and have Pando running and haven’t moved the file.
The software is straightforward and easy-to-use. The company stores files on its own supernodes for up to 14 days after the file is posted and recipients informed. Packages can be up to 1 GB in size during the beta period. (The implication is that limit will be raised when the beta period is over, but the company hasn’t promised that.) After that 14-day period, packages are still retrievable from any Pando clients that are running and have the file still available.
This is an interesting twist, because you could send out a large file – say, a video you’d created – to hundreds of recipients, all of whom would benefit from the initial high-speed availability on Pando’s servers and the swarming effect of many downloaders with Pando’s client. However, after 14 days, not only would Pando drop the file and thus reduce some large potential bandwidth costs, but for most downloads of this sort, most of the other recipients would have moved on and probably stopped participating in the swarm, too.
Pando’s encryption model is slightly more robust than Civil Netizen’s. The developers have chosen to use the 256-bit version of AES, the company said via email, although its FAQ states that Pando uses 128-bit encryption. All other communication between the Pando client and the company’s servers are conducted using certificate-authority validated SSL/TLS, which prevents tampering.
However, the .pando file is sent unencrypted and contains the security key necessary to decipher the retrieved file. Anyone with the .pando file would then be able to retrieve the same data with no additional validation or authentication.
Changing the World of Email Attachments — Where both Civil Netizen and Pando could shine is in bypassing the ugly world of one-off file transfers that currently use email attachments. While MIME (Multipurpose Internet Mail Extensions) has long made it relatively simple to send attachments reliably among varied email servers and clients, many companies that provide email services impose attachment size limits.
Many free email services allow you to send and receive total attachments per message of 2 MB to 10 MB. Some also have monthly, daily, or even hourly limits on attachments. Higher-end services have increased attachment limits over time to tens of megabytes, but even still, that’s a limit you must be aware of and track.
Plus, email servers aren’t designed well to handle large files. Some choke even when the attachment size is within limits. Retrieving a large file often takes substantially longer from a mail server than from a commensurate file server (even file server software running on the same hardware as the mail server software).
Pando has taken one step in that direction by planning a Microsoft Outlook 2003 plug-in that would enable Pando to be used as a substitute for large email attachments. Given the nature of both products, I would hope that plug-ins could be created for popular Macintosh email clients and other Windows email clients, too.
I’d also like to see designated recipients, so that when I need to transfer a file to, say, Adam Engst, I would drag a set of files onto an Adam icon on the Desktop or within the program. If Adam had pre-approved me, perhaps his copy of either package would automatically download the files I transmitted without further ado and alert him. This kind of trust could be made possible through these programs and the systems that support them, and would eliminate a lot of the fuss that file transfer places upon ordinary users.
All that said, when I wrote “Take Control of Sharing Files in Panther” and later “Take Control of Sharing Files in Tiger”, I found that the devil is in the details when it comes to making file sharing work. Whenever you have a group of people who need a common repository of files, file services like AppleShare, Samba, WebDAV, and FTP are still warranted. In these cases, you want a persistent set of consistently available, updated files found in the same place.
And for software companies or other organizations that need to distribute large amounts of files or a few large files, FTP and HTTP downloads still make the most sense, because there’s little chance except during new releases of having the right threshold of users downloading and retaining a file to get the benefit of the swarm behavior of P2P that Pando can leverage.
However, there’s a great place for this new method of bypassing all current forms of repositories and P2P, and, in the process, increasing the efficiency of retrieving files and reducing associated frustration.