One thing to be aware of, these replacement zlib libraries are not always "drop in" safe as some software such as nginx pre-allocate the zlib state buffers by hard coding in the original zlib structure sizes (the original zlib makes guarantees about memory usage that technically makes this behavior somewhat OK). Depending on how well the software is designed, the forked zlib versions may result in buffer overflows or other crashing. (nginx is fine - it spams your log file with errors and re-allocates).
I'll be watching this with some interest. A couple years ago I tried using some of the x86_64 assembly versions of inflate_fast() and longest_match() but found them to be subtly buggy -- they'd both sometimes do reads outside of the allocated memory and could crash if that happened to land at the end of a mmap'ed segment. I emailed all of the relevant authors, but nobody seemed to be maintaining that stuff any more. Honestly, just having a zlib with a bug tracker is a huge improvement.
I am sad to see that a lot of the zlib-ng seem to be around code style updates, which will make it hard for things to travel back and forth between the codebases. I guess one branch or another will eventually have to "win", which is sad.
Also, is zlib-ng going to commit to keeping the same zlib license? Some of the zlib improvements floating around on the net have snuck in more restrictive licensing. It would be fantastic if zlib-ng could hold the line on license creep.
The zlib code has to make numerous workarounds for old compilers that do not understand ANSI-C or to accommodate systems with limitations such as operating in a 16-bit environment.
This is true for many older open source libraries. I wonder:
- Are old platforms still tested as time passes? Is it really worth it maintaining extra cruft for old systems even if it is unknown whether it still works? Are there projects running CI for OS/2 Warp, MS-DOS, or A/UX?
- How many users of old or obscure platforms are there still around? Are they even updating such libraries?
- What platforms do we need to get rid of to get rid of autoconf/automake/configure and use e.g. CMake instead?
A lot even limit their codebases to C89, which I don't understand. Are there really that many platforms/compilers that don't effectively support C99 or greater? Is sixteen years not long enough for a language feature set to stabilize?
Maybe there are tradeoffs or platforms I'm just not aware of, but I at least tend to use C99 as the LCD.
Having code that's easy to integrate with existing projects is definitely a really good thing, and I honestly don't like to neglect Windows, but with all due respect, the platform kind of does it to itself.
C11 would be really nice to see as an LCD, especially for threading (or even just a native POSIX threads implementation).
I was more thinking of language features. threads.h would be nice, but, as you may know, despite it being a pretty simple API, neither glibc nor OS X libc has implemented it - there's only musl and then compatibility shims. Sigh. You could blame a lack of demand, but I'd call that chicken and egg: the whole advantage of it over pthreads in theory is portability due to standardization, but as long as the portability remains theory, so does the advantage.
> What platforms do we need to get rid of to get rid of autoconf/automake/configure and use e.g. CMake instead?
Why would you want that? Autotools is the devil we know. CMake has its own problems and from a statistical sample of one project implementing both, I came to the conclusion that CMake is more frustrating than Autotools when you need to do something unusual.
Exactly. autotools is exceptionally well designed for dealing with more exotic use cases like cross compiling. When designing a cross-compiling linux distribution(http://exherbo.org/docs/multiarch.txt) we discovered that autotools is by far the easiest to deal with when it comes to arbitrary host & build targets, compilers, etc. I would be even bolder and say that:
Those who do not understand autotools are doomed to reinvent it, poorly.
Exactly. autotools is exceptionally well designed for dealing with more exotic use cases like cross compiling.
...and it cannot deal with non-exotic use cases like non-UNIX environments (think Windows + Visual Studio) well at all. In contrast, with CMake you can generate an nmake file or Visual Studio project with one command.
Just like someone under unix could always just use mono's xbuild to build a project using visual studio's .vsproj files. Course, people tend to get grumpy when you tell them that they have to jump through hoops to use your project, just like your suggestion...
I use CMake to cross-compile firmware for my home automation hardware. It doesn't seem too bad, just a single file shared across all projects that defines the compiler binaries and settings. I like it much better than autotools.
Note that autotools handles all that (referencing also your link in another child comment) for you with no extra work. Writing a autoconf script for a native architecture means your project should compile for a cross arch.
It is for that reason I much prefer autotools, especially at the distribution level where you really don't want to be patching things. CMake can work, but I have had to do more patching for CMake build systems than I have had to for autotools; on that basis I consider autotools "better", at least for our use case.
I came to the conclusion that CMake is more frustrating than Autotools when you need to do something unusual.
Unusual in terms of...? If I have learnt anything from Go, Java (per Maven), etc. it's that the far majority of project don't need to do something unusual.
Conversely, if you give people the room to do unusual things, they will.
Unfortunately the minigzip implementation used is using a very minimal codebase to implement the functionality, among these is a problem where it will read and decompress 1 byte per call to inflate(). A patch and pull request for this has been available since yesterday at least, and I merged this today.
In stock zlib this codepath is only used if you defined Z_SOLO, but in zlib-ng it is the default unless you run configure with --zlib-compat (or otherwise set the correct defines).
Please retest with the updated commit, or even better with --zlib-compat if you want to get close to an apples-to-apples comparison.
EDIT:
PS: This goes to show that minigzip can not really be used as a reliable benchmark, since actual applications linking to the library could be using it differently (for better or worse).
As I mentioned in another comment, the minigzip used is very suboptimal due to reading and decompressing only 1 byte per inflate() call. Please retest with --zlib-compat or the fix that was merged today.
I'm not in the habit of including compression level 6 in the results, since the application I care about defaults to 5 :) And I wanted a good spread of different compression levels in the results.
If you want more performance out of zlib, step one is recompile.
I've seen flabbergasting speedup out of basic CPU-bound utilities simply by recompiling the source code for a fifteen-year-old binary using a modern compiler.
Which also brings up the point that if you want to benchmark zlib-ng, compile zlib with the same compiler.
This makes me wonder what a good strategy would be for deprecating old architectures. Remove support for them in new versions and backport bugfixes to the last version supporting the old architectures indefinitely?
It's a more difficult issue with something like OpenSSL as new features are added. Zlib is done though, there is zero interest in sdding deflate64 or a different algorithm, they could just put a version on ice, keep it around for distribution and make a more clean 2.0
