Chapter 27 deals with iostreams and all their subcomponents - and extensions. All kinds of fun stuff. -
- - - -So you want to copy a file quickly and easily, and most important, - completely portably. And since this is C++, you have an open - ifstream (call it IN) and an open ofstream (call it OUT): -
-
- #include <fstream>
-
- std::ifstream IN ("input_file");
- std::ofstream OUT ("output_file");
- Here's the easiest way to get it completely wrong: -
-- OUT << IN;-
For those of you who don't already know why this doesn't work - (probably from having done it before), I invite you to quickly - create a simple text file called "input_file" containing - the sentence -
-- The quick brown fox jumped over the lazy dog.-
surrounded by blank lines. Code it up and try it. The contents - of "output_file" may surprise you. -
-Seriously, go do it. Get surprised, then come back. It's worth it. -
-The thing to remember is that the basic_[io]stream classes
- handle formatting, nothing else. In particular, they break up on
- whitespace. The actual reading, writing, and storing of data is
- handled by the basic_streambuf family. Fortunately, the
- operator<< is overloaded to take an ostream and
- a pointer-to-streambuf, in order to help with just this kind of
- "dump the data verbatim" situation.
-
Why a pointer to streambuf and not just a streambuf? Well,
- the [io]streams hold pointers (or references, depending on the
- implementation) to their buffers, not the actual
- buffers. This allows polymorphic behavior on the part of the buffers
- as well as the streams themselves. The pointer is easily retrieved
- using the rdbuf() member function. Therefore, the easiest
- way to copy the file is:
-
- OUT << IN.rdbuf();-
So what was happening with OUT<<IN? Undefined - behavior, since that particular << isn't defined by the Standard. - I have seen instances where it is implemented, but the character - extraction process removes all the whitespace, leaving you with no - blank lines and only "Thequickbrownfox...". With - libraries that do not define that operator, IN (or one of IN's - member pointers) sometimes gets converted to a void*, and the output - file then contains a perfect text representation of a hexidecimal - address (quite a big surprise). Others don't compile at all. -
-Also note that none of this is specific to o*f*streams. - The operators shown above are all defined in the parent - basic_ostream class and are therefore available with all possible - descendents. -
-Return to top of page or - to the FAQ. -
- -First, are you sure that you understand buffering? Particularly - the fact that C++ may not, in fact, have anything to do with it? -
-The rules for buffering can be a little odd, but they aren't any - different from those of C. (Maybe that's why they can be a bit - odd.) Many people think that writing a newline to an output - stream automatically flushes the output buffer. This is true only - when the output stream is, in fact, a terminal and not a file - or some other device -- and that may not even be true - since C++ says nothing about files nor terminals. All of that is - system-dependent. (The "newline-buffer-flushing only occurring - on terminals" thing is mostly true on Unix systems, though.) -
-Some people also believe that sending endl down an
- output stream only writes a newline. This is incorrect; after a
- newline is written, the buffer is also flushed. Perhaps this
- is the effect you want when writing to a screen -- get the text
- out as soon as possible, etc -- but the buffering is largely
- wasted when doing this to a file:
-
- output << "a line of text" << endl; - output << some_data_variable << endl; - output << "another line of text" << endl;-
The proper thing to do in this case to just write the data out - and let the libraries and the system worry about the buffering. - If you need a newline, just write a newline: -
-- output << "a line of text\n" - << some_data_variable << '\n' - << "another line of text\n";-
I have also joined the output statements into a single statement. - You could make the code prettier by moving the single newline to - the start of the quoted text on the thing line, for example. -
-If you do need to flush the buffer above, you can send an
- endl if you also need a newline, or just flush the buffer
- yourself:
-
- output << ...... << flush; // can use std::flush manipulator - output.flush(); // or call a member fn-
On the other hand, there are times when writing to a file should - be like writing to standard error; no buffering should be done - because the data needs to appear quickly (a prime example is a - log file for security-related information). The way to do this is - just to turn off the buffering before any I/O operations at - all have been done, i.e., as soon as possible after opening: -
-
- std::ofstream os ("/foo/bar/baz");
- std::ifstream is ("/qux/quux/quuux");
- int i;
-
- os.rdbuf()->pubsetbuf(0,0);
- is.rdbuf()->pubsetbuf(0,0);
- ...
- os << "this data is written immediately\n";
- is >> i; // and this will probably cause a disk read
- Since all aspects of buffering are handled by a streambuf-derived
- member, it is necessary to get at that member with rdbuf().
- Then the public version of setbuf can be called. The
- arguments are the same as those for the Standard C I/O Library
- function (a buffer area followed by its size).
-
A great deal of this is implementation-dependent. For example,
- streambuf does not specify any actions for its own
- setbuf()-ish functions; the classes derived from
- streambuf each define behavior that "makes
- sense" for that class: an argument of (0,0) turns off buffering
- for filebuf but has undefined behavior for its sibling
- stringbuf, and specifying anything other than (0,0) has
- varying effects. Other user-defined class derived from streambuf can
- do whatever they want. (For filebuf and arguments for
- (p,s) other than zeros, libstdc++ does what you'd expect:
- the first s bytes of p are used as a buffer,
- which you must allocate and deallocate.)
-
A last reminder: there are usually more buffers involved than - just those at the language/library level. Kernel buffers, disk - buffers, and the like will also have an effect. Inspecting and - changing those are system-dependent. -
-Return to top of page or - to the FAQ. -
- -The first and most important thing to remember about binary I/O is
- that opening a file with ios::binary is not, repeat
- not, the only thing you have to do. It is not a silver
- bullet, and will not allow you to use the <</>>
- operators of the normal fstreams to do binary I/O.
-
Sorry. Them's the breaks. -
-This isn't going to try and be a complete tutorial on reading and - writing binary files (because "binary" - covers a lot of ground), but we will try and clear - up a couple of misconceptions and common errors. -
-First, ios::binary has exactly one defined effect, no more
- and no less. Normal text mode has to be concerned with the newline
- characters, and the runtime system will translate between (for
- example) '\n' and the appropriate end-of-line sequence (LF on Unix,
- CRLF on DOS, CR on Macintosh, etc). (There are other things that
- normal mode does, but that's the most obvious.) Opening a file in
- binary mode disables this conversion, so reading a CRLF sequence
- under Windows won't accidentally get mapped to a '\n' character, etc.
- Binary mode is not supposed to suddenly give you a bitstream, and
- if it is doing so in your program then you've discovered a bug in
- your vendor's compiler (or some other part of the C++ implementation,
- possibly the runtime system).
-
Second, using << to write and >> to
- read isn't going to work with the standard file stream classes, even
- if you use skipws during reading. Why not? Because
- ifstream and ofstream exist for the purpose of formatting,
- not reading and writing. Their job is to interpret the data into
- text characters, and that's exactly what you don't want to happen
- during binary I/O.
-
Third, using the get() and put()/write() member
- functions still aren't guaranteed to help you. These are
- "unformatted" I/O functions, but still character-based.
- (This may or may not be what you want, see below.)
-
Notice how all the problems here are due to the inappropriate use - of formatting functions and classes to perform something - which requires that formatting not be done? There are a - seemingly infinite number of solutions, and a few are listed here: -
-mmap()
- the file and copy the structure." Well, this is easy to
- make work, and easy to break, and is pretty equivalent to
- using ::read() and ::write() directly, and
- makes no use of the iostream library at all...
- How to go about using streambufs is a bit beyond the scope of this - document (at least for now), but while streambufs go a long way, - they still leave a couple of things up to you, the programmer. - As an example, byte ordering is completely between you and the - operating system, and you have to handle it yourself. -
-Deriving a streambuf or filebuf
- class from the standard ones, one that is specific to your data
- types (or an abstraction thereof) is probably a good idea, and
- lots of examples exist in journals and on Usenet. Using the
- standard filebufs directly (either by declaring your own or by
- using the pointer returned from an fstream's rdbuf())
- is certainly feasible as well.
-
One area that causes problems is trying to do bit-by-bit operations
- with filebufs. C++ is no different from C in this respect: I/O
- must be done at the byte level. If you're trying to read or write
- a few bits at a time, you're going about it the wrong way. You
- must read/write an integral number of bytes and then process the
- bytes. (For example, the streambuf functions take and return
- variables of type int_type.)
-
Another area of problems is opening text files in binary mode. - Generally, binary mode is intended for binary files, and opening - text files in binary mode means that you now have to deal with all of - those end-of-line and end-of-file problems that we mentioned before. - An instructive thread from comp.lang.c++.moderated delved off into - this topic starting more or less at - this - article and continuing to the end of the thread. (You'll have to - sort through some flames every couple of paragraphs, but the points - made are good ones.) -
- -Stringstreams (defined in the header <sstream>)
- are in this author's opinion one of the coolest things since
- sliced time. An example of their use is in the Received Wisdom
- section for Chapter 21 (Strings),
- describing how to
- format strings.
-
The quick definition is: they are siblings of ifstream and ofstream,
- and they do for std::string what their siblings do for
- files. All that work you put into writing << and
- >> functions for your classes now pays off
- again! Need to format a string before passing the string
- to a function? Send your stuff via << to an
- ostringstream. You've read a string as input and need to parse it?
- Initialize an istringstream with that string, and then pull pieces
- out of it with >>. Have a stringstream and need to
- get a copy of the string inside? Just call the str()
- member function.
-
This only works if you've written your
- <</>> functions correctly, though,
- and correctly means that they take istreams and ostreams as
- parameters, not ifstreams and ofstreams. If they
- take the latter, then your I/O operators will work fine with
- file streams, but with nothing else -- including stringstreams.
-
If you are a user of the strstream classes, you need to update
- your code. You don't have to explicitly append ends to
- terminate the C-style character array, you don't have to mess with
- "freezing" functions, and you don't have to manage the
- memory yourself. The strstreams have been officially deprecated,
- which means that 1) future revisions of the C++ Standard won't
- support them, and 2) if you use them, people will laugh at you.
-
Creating your own stream buffers for I/O can be remarkably easy. - If you are interested in doing so, we highly recommend two very - excellent books: - Standard C++ - IOStreams and Locales by Langer and Kreft, ISBN 0-201-18395-1, and - The C++ Standard Library - by Nicolai Josuttis, ISBN 0-201-37926-0. Both are published by - Addison-Wesley, who isn't paying us a cent for saying that, honest. -
-Here is a simple example, io/outbuf1, from the Josuttis text. It - transforms everything sent through it to uppercase. This version - assumes many things about the nature of the character type being - used (for more information, read the books or the newsgroups): -
-
- #include <iostream>
- #include <streambuf>
- #include <locale>
- #include <cstdio>
-
- class outbuf : public std::streambuf
- {
- protected:
- /* central output function
- * - print characters in uppercase mode
- */
- virtual int_type overflow (int_type c) {
- if (c != EOF) {
- // convert lowercase to uppercase
- c = std::toupper(static_cast<char>(c),getloc());
-
- // and write the character to the standard output
- if (putchar(c) == EOF) {
- return EOF;
- }
- }
- return c;
- }
- };
-
- int main()
- {
- // create special output buffer
- outbuf ob;
- // initialize output stream with that output buffer
- std::ostream out(&ob);
-
- out << "31 hexadecimal: "
- << std::hex << 31 << std::endl;
- return 0;
- }
-
- Try it yourself! More examples can be found in 3.1.x code, in
- include/ext/*_filebuf.h.
-
Towards the beginning of February 2001, the subject of - "binary" I/O was brought up in a couple of places at the - same time. One notable place was Usenet, where James Kanze and - Dietmar Kühl separately posted articles on why attempting - generic binary I/O was not a good idea. (Here are copies of - Kanze's article and - Kühl's article.) -
-Briefly, the problems of byte ordering and type sizes mean that
- the unformatted functions like ostream::put() and
- istream::get() cannot safely be used to communicate
- between arbitrary programs, or across a network, or from one
- invocation of a program to another invocation of the same program
- on a different platform, etc.
-
The entire Usenet thread is instructive, and took place under the - subject heading "binary iostreams" on both comp.std.c++ - and comp.lang.c++.moderated in parallel. Also in that thread, - Dietmar Kühl mentioned that he had written a pair of stream - classes that would read and write XDR, which is a good step towards - a portable binary format. -
- -It sounds like a flame on C, but it isn't. Really. Calm down. - I'm just saying it to get your attention. -
-Because the C++ library includes the C library, both C-style and - C++-style I/O have to work at the same time. For example: -
-
- #include <iostream>
- #include <cstdio>
-
- std::cout << "Hel";
- std::printf ("lo, worl");
- std::cout << "d!\n";
-
- This must do what you think it does. -
-Alert members of the audience will immediately notice that buffering - is going to make a hash of the output unless special steps are taken. -
-The special steps taken by libstdc++, at least for version 3.0,
- involve doing very little buffering for the standard streams, leaving
- most of the buffering to the underlying C library. (This kind of
- thing is tricky to get right.)
- The upside is that correctness is ensured. The downside is that
- writing through cout can quite easily lead to awful
- performance when the C++ I/O library is layered on top of the C I/O
- library (as it is for 3.0 by default). Some patches have been applied
- which improve the situation for 3.1.
-
However, the C and C++ standard streams only need to be kept in sync - when both libraries' facilities are in use. If your program only uses - C++ I/O, then there's no need to sync with the C streams. The right - thing to do in this case is to call -
-- #include any of the I/O headers such as ios, iostream, etc - - std::ios::sync_with_stdio(false); --
You must do this before performing any I/O via the C++ stream objects.
- Once you call this, the C++ streams will operate independently of the
- (unused) C streams. For GCC 3.x, this means that cout and
- company will become fully buffered on their own.
-
Note, by the way, that the synchronization requirement only applies to
- the standard streams (cin, cout,
- cerr,
- clog, and their wide-character counterparts). File stream
- objects that you declare yourself have no such requirement and are fully
- buffered.
-
I'll assume that you have already read the - general notes on library threads, - and the - notes on threaded container - access (you might not think of an I/O stream as a container, but - the points made there also hold here). If you have not read them, - please do so first. -
-This gets a bit tricky. Please read carefully, and bear with me. -
-As described here, a wrapper
- type called __basic_file provides our abstraction layer
- for the std::filebuf classes. Nearly all decisions dealing
- with actual input and output must be made in __basic_file.
-
A generic locking mechanism is somewhat in place at the filebuf layer, - but is not used in the current code. Providing locking at any higher - level is akin to providing locking within containers, and is not done - for the same reasons (see the links above). -
-The __basic_file type is simply a collection of small wrappers around
- the C stdio layer (again, see the link under Structure). We do no
- locking ourselves, but simply pass through to calls to fopen,
- fwrite, and so forth.
-
So, for 3.0, the question of "is multithreading safe for I/O" - must be answered with, "is your platform's C library threadsafe - for I/O?" Some are by default, some are not; many offer multiple - implementations of the C library with varying tradeoffs of threadsafety - and efficiency. You, the programmer, are always required to take care - with multiple threads. -
-(As an example, the POSIX standard requires that C stdio FILE*
- operations are atomic. POSIX-conforming C libraries (e.g, on Solaris
- and GNU/Linux) have an internal mutex to serialize operations on
- FILE*s. However, you still need to not do stupid things like calling
- fclose(fs) in one thread followed by an access of
- fs in another.)
-
So, if your platform's C library is threadsafe, then your
- fstream I/O operations will be threadsafe at the lowest
- level. For higher-level operations, such as manipulating the data
- contained in the stream formatting classes (e.g., setting up callbacks
- inside an std::ofstream), you need to guard such accesses
- like any other critical shared resource.
-
As already mentioned here, a - second choice is available for I/O implementations: libio. This is - disabled by default, and in fact will not currently work due to other - issues. It will be revisited, however. -
-The libio code is a subset of the guts of the GNU libc (glibc) I/O
- implementation. When libio is in use, the __basic_file
- type is basically derived from FILE. (The real situation is more
- complex than that... it's derived from an internal type used to
- implement FILE. See libio/libioP.h to see scary things done with
- vtbls.) The result is that there is no "layer" of C stdio
- to go through; the filebuf makes calls directly into the same
- functions used to implement fread, fwrite,
- and so forth, using internal data structures. (And when I say
- "makes calls directly," I mean the function is literally
- replaced by a jump into an internal function. Fast but frightening.
- *grin*)
-
Also, the libio internal locks are used. This requires pulling in - large chunks of glibc, such as a pthreads implementation, and is one - of the issues preventing widespread use of libio as the libstdc++ - cstdio implementation. -
-But we plan to make this work, at least as an option if not a future - default. Platforms running a copy of glibc with a recent-enough - version will see calls from libstdc++ directly into the glibc already - installed. For other platforms, a copy of the libio subsection will - be built and included in libstdc++. -
-Don't forget that other cstdio implemenations are possible. You could - easily write one to perform your own forms of locking, to solve your - "interesting" problems. -
- -To minimize the time you have to wait on the compiler, it's good to - only include the headers you really need. Many people simply include - <iostream> when they don't need to -- and that can penalize - your runtime as well. Here are some tips on which header to use - for which situations, starting with the simplest. -
-<iosfwd> should be included whenever you simply - need the name of an I/O-related class, such as - "ofstream" or "basic_streambuf". Like the name - implies, these are forward declarations. (A word to all you fellow - old school programmers: trying to forward declare classes like - "class istream;" won't work. Look in the iosfwd header if - you'd like to know why.) For example, -
-
- #include <iosfwd>
-
- class MyClass
- {
- ....
- std::ifstream input_file;
- };
-
- extern std::ostream& operator<< (std::ostream&, MyClass&);
-
- <ios> declares the base classes for the entire - I/O stream hierarchy, std::ios_base and std::basic_ios<charT>, the - counting types std::streamoff and std::streamsize, the file - positioning type std::fpos, and the various manipulators like - std::hex, std::fixed, std::noshowbase, and so forth. -
-The ios_base class is what holds the format flags, the state flags, - and the functions which change them (setf(), width(), precision(), - etc). You can also store extra data and register callback functions - through ios_base, but that has been historically underused. Anything - which doesn't depend on the type of characters stored is consolidated - here. -
-The template class basic_ios is the highest template class in the - hierarchy; it is the first one depending on the character type, and - holds all general state associated with that type: the pointer to the - polymorphic stream buffer, the facet information, etc. -
-<streambuf> declares the template class - basic_streambuf, and two standard instantiations, streambuf and - wstreambuf. If you need to work with the vastly useful and capable - stream buffer classes, e.g., to create a new form of storage - transport, this header is the one to include. -
-<istream>/<ostream> are - the headers to include when you are using the >>/<< - interface, or any of the other abstract stream formatting functions. - For example, -
-
- #include <istream>
-
- std::ostream& operator<< (std::ostream& os, MyClass& c)
- {
- return os << c.data1() << c.data2();
- }
-
- The std::istream and std::ostream classes are the abstract parents of - the various concrete implementations. If you are only using the - interfaces, then you only need to use the appropriate interface header. -
-<iomanip> provides "extractors and inserters
- that alter information maintained by class ios_base and its dervied
- classes," such as std::setprecision and std::setw. If you need
- to write expressions like os << setw(3); or
- is >> setbase(8);, you must include <iomanip>.
-
<sstream>/<fstream> - declare the six stringstream and fstream classes. As they are the - standard concrete descendants of istream and ostream, you will already - know about them. -
-Finally, <iostream> provides the eight standard - global objects (cin, cout, etc). To do this correctly, this header - also provides the contents of the <istream> and <ostream> - headers, but nothing else. The contents of this header look like -
-
- #include <ostream>
- #include <istream>
-
- namespace std
- {
- extern istream cin;
- extern ostream cout;
- ....
-
- // this is explained below
- static ios_base::Init __foo; // not its real name
- }
-
- Now, the runtime penalty mentioned previously: the global objects - must be initialized before any of your own code uses them; this is - guaranteed by the standard. Like any other global object, they must - be initialized once and only once. This is typically done with a - construct like the one above, and the nested class ios_base::Init is - specified in the standard for just this reason. -
-How does it work? Because the header is included before any of your - code, the __foo object is constructed before any of - your objects. (Global objects are built in the order in which they - are declared, and destroyed in reverse order.) The first time the - constructor runs, the eight stream objects are set up. -
-The static keyword means that each object file compiled
- from a source file containing <iostream> will have its own
- private copy of __foo. There is no specified order
- of construction across object files (it's one of those pesky NP
- problems that make life so interesting), so one copy in each object
- file means that the stream objects are guaranteed to be set up before
- any of your code which uses them could run, thereby meeting the
- requirements of the standard.
-
The penalty, of course, is that after the first copy of - __foo is constructed, all the others are just wasted - processor time. The time spent is merely for an increment-and-test - inside a function call, but over several dozen or hundreds of object - files, that time can add up. (It's not in a tight loop, either.) -
-The lesson? Only include <iostream> when you need to use one of - the standard objects in that source file; you'll pay less startup - time. Only include the header files you need to in general; your - compile times will go down when there's less parsing work to do. -
- - - - --See license.html for copying conditions. -Comments and suggestions are welcome, and may be sent to -the libstdc++ mailing list. -
- - - - - -