I am using a custom installer, and am not using cloud-init with it. Instead, I'd like to put something else into user-data: a Linux initrd image, that my netboot/iPXE script can download at install time.
The image holds the data needed to complete an automated/unattended install (i.e., Debian preseed.cfg, SSH keys, first-boot scripts, etc., similar to cloud-init's use.) It's not very large: right now it's either 1.8 KB or 5.5 KB, depending on file format (*see below).
== Background ==
Modern Linux initrd's are cpio archives, + optional compression.
There are several cpio formats, but all of them are barebones/simplistic.
Linux supports 'newc' and 'crc' cpio formats; these are both ASCII formats.
(But, they do contain some ASCII control characters, like '\0'.)
Finally, cpio initrds are typically gzip-compressed; gzip output is *definitely* binary/8-bit.
== Problem ==
What I'm finding is that when I store a gzip'd cpio archive ("initrd.gz") in user-data, I cannot retrieve it from the meta-data service. Instead, it serves me a zero-length response.
When I store a plain cpio archive (not gzip'd), I am able to download that file from the meta-data service successfully.
That would be an acceptable workaround, except: Sometimes, when I include certain kinds of files in the archive, it becomes un-downloadable in exactly the same way as my initrd.cpio.gz is.
== Suspected Cause ==
I think the meta-data server is either choking on -- or being overly paranoid about -- binary/8-bit data in user-data. (The authenticated API doesn't seem to have a problem, and uses base64 for encapsulation; the web UI does OK with it too.)
== Additional Thoughts ==
If you find yourself tempted to think that maybe only plain/ASCII text is supported for use in user-data, and not binary/8-bit data, be aware of this: cloud-init explicitly supports several formats besides the usual #cloud-config
scripts people are familiar with, including multi-part MIME data, and gzip-compressed content.
So, even though I might be using the user-data service in a slightly unusual way, if that service cannot transport binary or gzip data, then it will also fail for people using typical/supported cloud-init features for their intended purpose. For that reason, I think this problem qualifies as a (minor?) infrastructure bug/defect, that you guys will want to resolve.
For more on cloud-init's use of binary user-data, see: https://cloudinit.readthedocs.org/en/latest/topics/format.html
Thank you for your time,