First off, a heartfelt thank you to NY.pm and mongoDB for giving me the opportunity to talk a little about my adventures in compiling perl with Microsoft Visual Studio 2013 Community Edition.
The process of putting together the talk lead me to discover another two more of those pesky bugs due to platform differences in handling files. These are more interesting than the directory separator character style issues that I have been ranting about, mostly because of how hard they were to see just by looking at the code. I will submit patches, obviously, but my hope with these blog posts is to re-emphasize that these platform-specific differences do exist, whether we like it or not, and if more people are aware, fewer bugs will make it into CPAN modules in the first place.
Bug 1: Corrupt images with App::revealup
I decided to put together the presentation using App::revealup. As I was reviewing my presentation, I noticed various screenshots would not display properly, or not at all. I noticed that PNG images were not displaying. It didn’t matter what browser I was using. It also did not matter whether I was using Strawberry’s perl, or the perl I had built. I could not figure out the problem, but I did realize JPGs were mostly displayed intact, if only with occasional corruption, and sometimes really visible artifacts.
The screenshots below show a PNG that does not display, a JPG that does not display, and the same image saved with a different compression level seemingly displaying perfectly fine:
There was one image, a screenshot of the Visual Studio’s GUI IDE window, which, no matter what I tried, did not display properly. There was always some visible problem with it. I tried several compression levels, and decided to stick with the one that looked the least bad.
I just could not figure out what was wrong, but I also wanted to finish the presentation.
About an hour or so before the talk, however, something occurred to me.
I decided to check what would happen if I used a Windows BMP. Compare the screenshots below, showing a JPG and a BMP, respectively:
By now, knowing what I know about file and compression formats, I was pretty certain I knew what was going on. As one final test, I decided to temporarily serve the exact same presentation from one of my Linode VPSs.
Absolutely no problems there.
Mystery solved. However, I had to go give my talk, and I did not have a chance to locate source of the problem in the module. I’ll come back to this at the end of the post.
For now, see if you can tell what’s going on.
Bug 2: Failing tests with Complete::Util
At the end of the talk, I decided to install a recently uploaded module from CPAN. After looking briefly at the list of recent uploads, I decided App::wordlist which looked innocent enough. Unfortunately, cpanm App::wordlist failed, due to a failing test in Complete::Util.
The test script complete_program.t tests the routine Complete::Util::complete_program. A brief look at both the module and the test script reveal a lot of opportunities for making sure platform specific paths don’t cause problems, but that is not where the bug lies. After all, passing Unix style paths to Windows APIs works with no problems.
The test fails because of the last check in this line:
push @res, $_ if $_ =~ $word_re && !(-d "$dir/$_") && (-x _);The test script creates the following files:
mkexe("$dir/dir1/prog1");
mkexe("$dir/dir1/prog2");
mkexe("$dir/dir2/prog3");sub mkexe { write_file($_[0], ""); chmod 0755, $_[0] }Here is what perldoc perlport has to say about that:
chmod
- Only good for changing “owner” read-write access, “group”, and “other” bits are meaningless. (Win32)
- Only good for changing “owner” and “other” read-write access. (RISC OS)
- Access permissions are mapped onto VOS access-control list changes. (VOS)
- The actual permissions set depend on the value of the CYGWIN in the SYSTEM environment settings. (Cygwin)
- Setting the exec bit on some locations (generally /sdcard) will return true but not actually set the bit. (Android)
On Windows, if you want to create a zero-byte executable, you might want to give it the .bat extension. As far I can see, that won’t cause any more portability problems than the module already might have, but at least, it won’t cause a spurious test failure on Windows.
Again, I will, of course submit a match in due course, but I also find it helpful to document the process of exploration. After all, I would like to have something to show for the effort of diving into someone else’s code, trying to figure out what is failing and why etc.
Back to App::revealup
I am assuming by now you have already realized that the images being served by App::revealup were being corrupted through CRLF-translation. That is why the bitmap file displays just with a color shift in a predefined area, that is why some JPEG files display with varying degrees of corruption, and some JPEG files don’t display at all, and that’s why PNG files uniformly fail to display.
Here is why that happens.
When App::revealup responds to your request, it does a few checks, and if the resources exists on the local filesystem, it returns the contents of said resource:
return App::revealup::util::path_to_res($path) if $path->exists;$path variable here is an instance of Path::Tiny which provides several slurp methods.
App::revealup uses the plain slurp:
sub path_to_res {
my $path = shift;
if( $path && $path->exists ) {
my $c = $path->slurp();
my $meta = ['Content-Length' => length $c ];
if( my $mime = MIME::Types->new->mimeTypeOf($path->basename) ){
push @$meta, ('Content-Type' => $mime->type );
}
return [200, $meta , [$c]];
}
return [404, [], ['not found.']];
}Path::Tiny documentation explains the slurp method really well.
Basically, the plain slurp call will do a:
local $/;
return scalar <$fh>;which will translate CRLF pairs to LF.
You can see that clearly when you compare the local file with the copy retrieved from the App::revealup server on Windows:
Checking RFC2616, we note the following:
3.7.1 Canonicalization and Text Defaults
When in canonical form, media subtypes of the “text” type use CRLF as the text line break. HTTP relaxes this requirement and allows the transport of text media with plain CR or LF alone representing a line break when it is done consistently for an entire entity-body. HTTP applications MUST accept CRLF, bare CR, and bare LF as being representative of a line break in text media received via HTTP.
So, even for text files, I see no reason App::revealup ought to provide line-ending translation. The correct fix is to replace line 23 in App::revealup::util:
my $c = $path->slurp();with
my $c = $path->slurp_raw();PS:
For reference: