Series: Document Collaboration
Now that the Internet lets you write anything with anyone, how do you actually do it?
Article 1 of 3 in series
One of the great features of the Internet, according to the pundits, is how wonderfully it enables communication and collaboration between widely separated individualsShow full article
One of the great features of the Internet, according to the pundits, is how wonderfully it enables communication and collaboration between widely separated individuals. Much of this communication occurs via email or, in some cases, via instant messaging, and works well. But I'll bet that if you've ever tried to collaborate with a group of people on the Internet to write and edit a document, you've found that moving the document back and forth across the Internet is merely a minor aspect of the entire process, and that effectively collaborating with people is quite difficult.
Here at TidBITS, we spend a lot of time together editing documents - even though none of us share a physical office - and we've worked out a system that has proven quite efficient for us. But since we all write for a variety of other publications and organizations, we've also come across numerous other collaboration approaches. It's continually astonishing to us just how hard it is to put together a good document collaboration strategy and how different groups have chosen to handle the problem.
This week I'll look at some of the major variables involved in any document collaboration system and give you a detailed look at exactly how we share documents within TidBITS. In the next installment, I'll explore a number of other systems I've used with other groups and talk about how well each worked. By the end, I hope you'll have a starting point from which to make your next document collaboration task fast and efficient.
Collaboration Design Variables -- There are several variables to consider whenever you're trying to set up a document collaboration process, especially one which doesn't rely on fancy tools that can be expensive, hard to deploy, and tricky to use. My assumption here is that you have a document you wish to have reviewed and edited by others; although documents may have multiple "owners" throughout a publication process, at any one time there should be only one person ultimately responsible for it.
Version control. Perhaps the most important consideration is how you'll handle version control, or making sure that everyone's comments and changes are integrated properly into the final document. There are two approaches here, scattershot and round-robin. In a scattershot approach, you send the document out to all the reviewers, then collect and combine those reviewer's comments. It's easy, requires little technology, and doesn't make demands on the reviewers. Unfortunately, it also often results in duplicate comments from reviewers (who also don't benefit from seeing each other's comments) and gives you a lot of extra work in evaluating and integrating all the different responses. That extra work can make for a better document in the end, though, since you maintain a coherent view of the whole. In the round-robin approach, each reviewer edits the document in turn, which lets them comment on one another's changes and eliminates some extra integration work. Unfortunately, you generally still have to do major cleanup on the final document. Worse, the round-robin approach takes time, and the amount of time goes up with the number of reviewers, since only one person can have the document at a time.
Number and type of reviewers. If you're passing a document to one other person and receiving it back, it shouldn't be hard to agree on an approach. However, as the number of reviewers increases, it's often best to choose one of three approaches: simple, rigid, or assisted. A very simple approach that requires little of the reviewers works best with ad-hoc reviewers, though it increases the work of the person in charge of the final document. A detailed approach with rigid rules about markup styles for comments and changes works better when you have a close-knit group that can agree on a process. Finally, if you simply can't get your reviewers to follow a process, assistance from document management tools may become necessary. I can't comment on these for the most part, since they tend to be large, complex, and expensive - three adjectives we at TidBITS avoid like the plague.
Document location. The resources available for document location and access - both for you and for your reviewers - play into how you set up your collaboration process. Centralized servers can work well, especially for a round-robin approach with a number of trained reviewers, but they're often hard to use and overkill for a couple of ad-hoc reviewers. In that case, a decentralized solution (usually using email, floppies, or CD-Rs to distribute files) usually proves more effective.
Document format. The document format you choose to work in can affect your collaboration process significantly, but it's important to remember that you can use different formats at different points in a publication process. For instance, even if an article is destined for QuarkXPress, that doesn't mean you can't design the review process to use Microsoft Word documents or even straight text files. Plus, programs like Word offer (occasionally inscrutable) collaboration tools of their own, whereas other formats may require you to decide on some simple markup to indicate additions, changes, deletions, and comments.
Document markup. No matter what format you choose (and different projects may call for different formats), it's a good idea to negotiate markup conventions ahead of time. Otherwise you'll waste a lot of time trying to figure out different markup approaches, particularly in situations where you'll be passing many documents by a number of reviewers. Even in Word - which has revision tracking and a comment feature that identifies the author of each change or comment - it's a good idea to agree on usage conventions ahead of time.
Keep in mind that no one collaboration strategy will fit every situation, and you may need to come up with several strategies for different groups.
TidBITS Collaboration -- We've adopted a specific strategy for document collaboration that works extremely well for us. It's not perfect, and it undoubtedly wouldn't work in all situations, but let me explain it so you can get a sense of how you might go about using parts of it in your own collaboration process.
We rely on a round-robin editing approach among a small number of editors working on a centralized server, accessible via both FTP and AppleShare over IP. All of our documents are in Nisus Writer, which provides styles for markup, although no revision tracking or comment features like Word's. Here's how a document might go through the entire process to end up in a TidBITS issue.
One of us creates the draft document, applying the styles necessary for our issue creation and distribution macros. That person also does an initial edit pass. We don't worry about small changes, but we mark meaningful changes with colors so others can see what was done. Additions or modifications appear in green, and proposed deletions appear in red. If we need to make a comment or query about a paragraph, we make the comment in red on a line by itself below the paragraph to avoid confusing the meaning of the paragraph with intratextual comments. All comments are signed with the initials of the person making the comment. If the comment or query applies only to a small bit of text, we mark that text in blue. Comments and text for deletion are both red because all red text is automatically removed just before distribution.
After finishing the initial edit pass, the document goes into a folder called IN on our server. Appended to its name are a version number and a set of initials. So this article is currently called Collaboration.1.ace, which indicates it has undergone one edit pass by me.
Let's say Jeff Carlson wants to take the next edit pass. Anything in IN is fair game, so he checks the document out by moving it (a Finder drag via AppleShare over IP, but possible in Interarchy via FTP with the Rename/Move command) to another folder called OUT. He also adds his initials to the filename so anyone trying to figure out who owns the document can tell by the fact that it's now called Collaboration.1.ace.jlc. Checking out the document also involves downloading for Jeff and Geoff, while I, since I'm on the same network as the server, just open it directly, although I otherwise follow the same rules. While Jeff has the document out, he can make any changes he wants, using the same colored markup scheme and comment approach outlined in the first step.
As Managing Editor, Jeff often has the task of sending an article back to the original author, if it wasn't written by one of the TidBITS staff members. Here's where our system falls down a bit. When copying text from Nisus Writer to any other program (such as Eudora), color information is lost (we also prefix all comments with three asterisks so they stand out even without colors). Interestingly, a similar color pasting problem afflicts Microsoft Word as well, so we can't export from Nisus Writer into Word as a workaround. And since relatively few people use Nisus Writer, sending the original file isn't generally a useful option. Thus, our preferred approach is to send the article back in the body of an email message and ask the author to make comments and offer suggested changes just as though they were replying to a normal email message. Jeff then incorporates the changes manually. That works well as long as the changes are relatively minimal, but for more significant rewrites we find that we just have to give the original file back to the author, let him or her make the necessary changes, and then restart the entire process. When Jeff''s done, he uploads the file to IN again, and changes the name to Collaboration.2.ace.jlc so we know it's gone through the second edit pass and who did it.
At any time during this process, we may send the article draft out to expert friends for quick technical review. They too get the article in the body of an email message and make comments and corrections in their reply. Whoever sent it out for review then has to incorporate the corrections and address the comments in the file, checking it out and back in again as necessary. At times, the list of initials at the end of the filename gets too long, at which point we delete some from the beginning of the list.
When he's ready, Jeff moves the current version of the article into OUT, copies all the text, complete with colors and comments, and pastes it into what we call a "copy file," which is the working draft of the full issue. Once the copy file holds the latest versions of all the articles, it too goes into IN and follows the same process rules, although we only add initials to the filename when it has been moved into OUT, since that's happening on Monday when we need to know exactly who has the file open at all times. On Mondays we also tend to use email and the phone fairly heavily to let others know when the document moves to avoid wasting time in between manual checks of IN and OUT.
Throughout all of this, colors and comments stay intact. At the last step before actual distribution, Geoff Duncan gives the issue the final read-through (often out loud - hopefully his neighbors don't mind) and deletes all the comments and text marked for deletion.
For the most part, our system relies on simple technology - an AppleShare/FTP server and colored text in a word processor. We intentionally try to keep our markup rules simple so there's never any confusion internally about what's happened. And when we bring in reviewers via email who don't know our approach, either they don't need to know it, or we can explain it easily.
However, our system also relies on having a small number of technically savvy reviewers with excellent attention to detail. Our approach would fall apart if anyone was overly sloppy or failed to follow the rules, especially those relating to checking files in and out, since we could end up overwriting someone else's changes. But there are other approaches that work better in such situations, and in the next installment, I'll look at some that I've used with varying degrees of success.
Article 2 of 3 in series
Last week I looked at some of the variables involved with creating document collaboration systems, and walked through the process that we use with TidBITSShow full article
Last week I looked at some of the variables involved with creating document collaboration systems, and walked through the process that we use with TidBITS. This week I'm going to look at several systems I've used personally. Discussions in TidBITS Talk have also introduced some tools with which I have no experience, so make sure to check out that hot topic for a more complete picture.
Hit the Books -- I've written books for a number of different publishers, and with one exception, they've used similar collaboration processes. In most cases, I'm a lone author working with one or two editors, which means sending chapters back and forth via email is all that's necessary. Sometimes publishers like to have draft chapters submitted via FTP, but that's proven more trouble than it's worth.
In each case, the publishers dictated using Microsoft Word as the document format. Although I wrote my first two books in Nisus Writer and exported to Word when I had to send them in, I've stuck with Word since. I far prefer writing in Nisus Writer, but the pain of exporting, importing, and massaging the text - often multiple times per chapter - overwhelmed my preferences.
Before Word 6, major changes to the text were marked with simple colors (never character styles, which could slip through to layout), but since then each project has relied on Word's revision tracking with mixed results. It's nice, particularly with multiple editors, to identify who made which changes when, but documents with revision tracking become messy quickly. You can hide some visual clutter by setting the style of deleted text to be Hidden rather than Strikethrough. Even still it's easy to cause confusion around paragraph breaks when adding and deleting text while using revision tracking, particularly when paragraph styles change at those points in the document. Plus, although Microsoft tries to make it easy to accept or reject changes, it's so much work to address a large number of changes in a book-length document that you tend to accept them all and move on.
Conventions for comments have also been similar between projects, with comments appearing on lines by themselves after the paragraph in question, usually in a special Comment paragraph style to differentiate them from the text. With multiple editors or reviewers, convention called for everyone to sign comments with initials. That approach has worked so well that even after the appearance of Word's comment features, I've never been asked to use them while writing a book. I suspect they're too unwieldy for serious use: - the yellow comment pop-ups are annoying, the comment pane is awkward and uses too-small text by default (and in Word 2001 on the Mac, the comment pane's Close button improperly requires two clicks to activate). Also, since Word insists comments be attached to at least one word, people tend to select lots of text for the comment, thus making large sections of a heavily commented document a bright yellow.
My Eudora Visual QuickStart Guide was an exception to the process above. It has relatively little text, many screenshots and captions, and a specific layout, all of which combined to make me do the writing and layout directly in QuarkXPress. Since QuarkXPress 3.3 lacks a good built-in way to make comments in files, I sent the files to my editor, who printed them out, marked them up with a red pencil, and sent them back via UPS. It sounds crude, but because so many comments related to layout (kerning, alignment, and so on), on-screen proofing probably would have been less accurate and more work.
Magazine & Web Writing -- Magazine and Web writing is similar to books in terms of the number of editors (one or two at most) and the methods of exchanging files (always email). Magazines also rely primarily on Word for their document format, although Web publications often prefer either straight text in the body of an email message or HTML.
With most periodicals, comments are relatively minimal because the articles and the deadlines are short. A longer piece for a monthly magazine is more likely to undergo significant editing and require multiple passes by different editors. One such publication had some conventions that had apparently evolved organically, since they had the right idea, but didn't go far enough. Changes either weren't marked at all, were marked in blue, or were handled via Word's revision tracking. Either of the last two would have worked fine, but the editors often weren't consistent, which writers found confusing.
Similarly, although some editors put their comments at the end of paragraphs, others inserted them right in the middle of the text. A few long comments inside a paragraph made it easy to lose track of the actual text. As with the book editors, no magazine editors I've worked with have ever used Word's comment feature. Some other situations I've experienced have been quite different, however.
Legal Documents -- When I helped start the XNS Public Trust Organization, the other directors and I worked on a number of long, complex documents - things like our charter, the XNS Global Terms, and various legal agreements. Although there were only four board members and our lawyer, we didn't have an author/editor situation, a centralized server, or reviewers who were publishing professionals.
The approach we used was to email Word documents around. Because of busy schedules, we were able to use a round-robin approach, so each of us was able to see the previous comments with the exception of our lawyer, who worked in parallel with the rest of us. (I tried to use Word's Compare Documents and Merge Documents features to merge his comments and changes with a complete lack of success.) We used revision tracking somewhat, but since we were mostly reviewing and discussing these documents, relatively few changes were necessary. However, we ended up using Word's comment feature heavily (about 100 comments over 2 documents), and although it worked acceptably, I found it clumsy for heavy use.
An unusual situation occurred when we had to finalize the XNS Global Terms just before launch. Delays meant we were working under a tight, one-day deadline, and we had about ten people involved, half of whom were lawyers. Initially we tried assigning the documents to a single person and hashing through remaining issues in a conference call, but that proved impractical. One of the lawyers recommended taking it back to email but keeping one person in charge of each document. Since the original documents were in Word, we kept them there, even though our eventual goal was to hand them off for conversion to HTML that evening. Plus, even though we were using a variety of Macs and PCs, everyone had a recent version of Word.
Word's revision tracking worked extremely well in this case, since it marked clearly what had been changed. We had no copy editors waiting at the end, so it was important to make sure that even small changes were marked. We used Word's comment features occasionally when remarking on something in the text that only the document owner needed to see, but we kept most of the commentary in the email messages used to carry the attached documents around. That let the discussion run fast and free (I set Eudora to check mail every minute) without forcing everyone to open and scan through each document. Putting comments between paragraphs, as I've done in most other projects, didn't happen, in part because people were afraid to add text to the document that might prove confusing to strip out later, and in part because we didn't have time to agree on comment conventions.
One lesson I've learned from these experiences is that the utility of Word documents is significantly hampered by having lots of user-defined styles in the document. Without a fair amount of care, not to mention knowledge of how Word styles work, a Word document rife with styles is a copy editing minefield just waiting for reviewers to walk through. The documents from these collaborations were fine in terms of content, but an utter mess otherwise. Some paragraphs were in the wrong styles, others appeared correct but had in fact been "fixed" manually, and almost anything Word did automatically (like numbered lists - a major portion of legal documents) was almost certain to be screwed up. Cleaning up the documents was easier than reformatting from scratch, but not much.
Simple Document Collaboration System -- To distill my experiences with the processes above and from the TidBITS system I described last week, here are six pieces of advice for creating a successful document collaboration system that revolves around files intended for some form of publication, be it a group term paper, team report, magazine article, or technical book.
Settle on a format before starting. In all likelihood, it will be Microsoft Word, but if you choose something else, make sure everyone has software that can open and edit the files. Pay special attention to cross-platform issues. Relying on other programs to import and export is often a significant obstacle.
Test your distribution mechanism before relying on it at deadline. Email attachments generally work well these days, but a test can save major headaches. See our series of articles explaining email attachments for assistance.
Agree on basic conventions for marking changes. Use either colors or Word's revision tracking to mark changes, or if your editing environment has no color capabilities, some sort of intratextual markup (but be sure to remove it at the end). Mark changes when they're at all major. With Word's revision tracking, it's often best to have Word hide deletions to make the document easier to read.
Agree on how comments will be made, either by putting them between paragraphs (in a different user-defined style or with some prefixed characters) or by using Word's comment feature. If you're using Word, make sure everyone enters their initials in the User Information tab of the Preferences so comments are identified, and if not, make sure everyone signs their comments.
Encourage consistency and adherence to agreed-upon rules. This is primarily important when multiple people are making roughly equal contributions to documents with lots of formatting. In situations where you just want comments or are working with minimally formatted documents, it's easier to let reviewers work however they want and integrate comments manually.
Make sure you know who will be responsible for assembling the final document, which will involve integration of comments, approving changes, removing all markup and comments, and general copy editing and document cleanup.
Going Online -- All of these systems discussed above used the Internet purely as a transport mechanism. In the next installment, I'll look at several services that offer collaboration environments via the Web with no need for special software or plug-ins of any sort.
Article 3 of 3 in series
Last week I examined a number of document collaboration systems I've used and passed on some advice for setting up a system of your own. However, all the systems I talked about involved sending files - usually Microsoft Word files - via the InternetShow full article
Last week I examined a number of document collaboration systems I've used and passed on some advice for setting up a system of your own. However, all the systems I talked about involved sending files - usually Microsoft Word files - via the Internet. This week I'm going to look at a couple of document collaboration systems that exist entirely on the Internet: QuickTopic and WikiWikiWebs. A few similar systems that I haven't used have also been mentioned in TidBITS Talk - be sure to check them out too.
QuickTopic Document Review -- The first of these services is the new QuickTopic Document Review Web site. First, QuickTopic itself is a lightweight Web-based single-topic discussion space. It's not completely private - the only security is through obscurity (the URL for your discussion contains a randomized set of letters and numbers) - but it could be useful for quick discussions. It displays messages one on top of the other on a Web page, and it can email you the messages if you prefer. It's a clever idea, but I prefer to use Eudora, which offers more flexible reading and writing environments, and archives everything for posterity.
I've found QuickTopic Document Review more interesting, because it provides an easy way to let any number of people read and comment on a document without worrying about file formats, file exchange, or comment conventions. All anyone needs is a Web browser.
Here's how it works. First, someone uploads an HTML document, and QuickTopic Document Review prefixes each paragraph with a "comment dot" and a sequentially numbered link. To make a comment, click the comment dot or numbered link, and then enter your comment in the text entry box that appears on the next page. When you submit the comment, QuickTopic creates a new discussion linked to the document and displays it. Paragraphs with comments get a little eyeglasses icon in the original document, next to the comment dot. You can also view comments alone or comments along with the paragraph they refer to, and you can sort comments by poster or by paragraph number, all of which can be helpful when integrating comments into your original document (which you must do manually, of course).
The XNSORG Communications Working Group recently tried QuickTopic Document Review for reviewing a large white paper. It worked well, though there are some rough spots. For instance, making comments and submitting them both take you to new pages, so you must either create comments in new windows or click Back twice to return to the original document. The developer of QuickTopic is constantly polishing the system, so I have high hopes for its future.
To get a sense of how QuickTopic Document Review works, try commenting on this article via the link below.
Wikis -- QuickTopic Document Review enables reviewing of documents, but what if you want multiple people to edit the same document online? For that, you can turn to another technology called a WikiWikiWeb (the term comes from the Hawaiian word for "quick" and is often abbreviated to "wiki").
A wiki is Web-based software that provides live document editing via the Web. In fact, a wiki enables live editing of entire Web sites, complete with automatic page creation. This level of freedom can be unsettling, accustomed as we are to the laborious process of creating, editing, uploading, and modifying Web pages. It's even more disturbing to think that anyone could edit or even delete your text. And yet, from what I can gather from reading wikis and talking to people who use them seriously, such textual vandalism seldom happens.
The main reason is akin to why open source works - social pressure to fit into the group. Other protection mechanisms exist as well. Every version of a wiki page is saved (efficiently, through the use of diffs that record just changes between versions) so you can always go back to a pre-vandalized version. Plus, some wikis provide IP-range and password security to individual pages so, for instance, everyone can read a page, but only some people can make changes. Another option lets only authorized people edit a page, but allows anyone to add comments to the end.
Another aspect of wikis that tends to throw people is that formatting is minimal. Wikis aim to promote communication, not layout. The level of formatting varies between wikis, but there are usually simple wiki-specific formatting rules (like turning lines starting with asterisks into bulleted lists, turning lines of several dashes into horizontal rules, and so on). Many wikis also support HTML markup, though most people don't bother since it's seldom worth the effort. Some wikis support graphics, but it's safe to say that most primarily contain text.
Using Wikis -- The original WikiWikiWeb was created by Ward Cunningham, but I've found that site's explanations overly abstract and disorganized. It's interesting to browse around in Ward's original wiki, though I'll warn you that it's easy to become brain-boggled in the process.
Since the wiki software itself is open source, a vast number of wiki clones have arisen, written in a wide variety of languages and running on a wide variety of platforms. Some even provide free wiki space to all comers, so you can set up your own wiki for testing or document collaboration, such as on the WikiWeb site (where they also sell wiki software for intranet use).
I used the WikiWeb site to collect questions for the annual Netters Dinner Survey at Macworld Expo this year. People could read the existing questions, and then either edit them (to add new answers, for instance) or add new questions. It worked relatively well for gathering a base set of questions which I used when emceeing the obligatory raise-your-hands survey. I've also set up another page on WikiWeb where I originally encouraged people from TidBITS Talk to explore and post comments - it showed me that wikis are too free-form for discussions, since comments can be added anywhere and there's no indication of when comments were made or (sometimes) who made them. Where wikis shine is creation and editing of documents, and so you can see what it would be like, I've uploaded this entire article to the WikiWeb site as well.
I've had more experience with XNSORG's internal wiki (we hope to open it up soon) for developing and reviewing content. Pages start in one of two ways - either essentially blank or relatively complete. For instance, we're talking about new content for our home page right now, so I created a page (page creation is merely a matter of naming the page and clicking a link) and roughed in some text. My goal is not to do all the work on the first pass myself, but to put enough material up there that others, when I send them the URL to that page, will be jogged into adding text, making changes, or perhaps just making comments. Plus, I can make changes any time I want without needing to distribute new versions. This approach can work well for starting a document no one desperately wants to create, since it spreads the workload. In this case, someone always ends up taking responsibility for the document, which entails copy editing, checking links, and tweaking the formatting.
At other times it makes more sense to take a mostly completed document and put it up on a wiki page for review and editing. This approach can save time for the person primarily responsible for the document, since the people reviewing often find and fix minor errors and can add or modify explanations as appropriate. This doesn't eliminate the need for the author to do the final edit pass, but it remains a flexible way to collect comments and changes without maintaining or distributing multiple versions.
Other uses we've found for the wiki include an easy way to publish working group meeting minutes without the effort of building and posting traditional Web pages, an open agenda to which all members of a working group can add items, "road map" documents which change frequently, and to-do lists for members of working groups. Overall, I'd say the wiki is a great success, and we're still thinking of ways to use it. For instance, once we can open it up to the public, we'd like to use it for FAQs, so people could ask questions and we could answer them, right on the same page as all the others while avoiding the slow process of revising and posting traditional HTML documents in a group setting.
Wiki on the Mac -- Although I found the public WikiWeb site, I wanted to see about running a wiki on a Mac. Most wikis are written in nominally cross-platform languages like Java or Perl, but I have neither the time nor the expertise to get them running on a Mac. Then I was alerted to the existence of Swiki, which was written in a Smalltalk variant called Squeak, developed initially as a research project at Apple (the developers reportedly followed Alan Kay to Disney). Swiki (I can't believe they missed naming it SqWiki, or "squeaky") requires its own Web server, called Comanche, and you can download and install the complete package for free. It wasn't hard - just follow the instructions on the second link below (since the instructions themselves are on a wiki, I did some editing in the one place I found them confusing).
I haven't tested it under any strain, but it seems to work on my PowerBook G3 (other than email notification of changed pages). I've also been impressed with Swiki's feature set - although I'm by no means experienced with wikis, Swiki seems to offer significantly more features in terms of formatting and access controls than many other implementations. If you like playing with Internet servers, Swiki is definitely worth a look.
Although the PowerBook is a lousy server machine for the simple reason that I take it offline on occasion, I'm considering testing Swiki as a way to collaborate with authors on TidBITS article drafts. The two main drawbacks are that editing tools available in a Web browser text field are primitive at best, and we'd lose our internal color coding system if we took the text out of Nisus Writer. However, I think I can address these criticisms with a macro that uses Nisus Writer as the editing environment and translates between Nisus Writer's formatting and Swiki markup.
We'll see how it works, and perhaps I'll end up with an additional resource in my document collaboration tool kit for future projects. In the meantime, I hope this series has been helpful in providing ideas for your own collaboration needs, and make sure to share other approaches you've used on TidBITS Talk.