# Saving strings/objects in TAP results to FITS **URL:** \ **Category:** Rubin Science Platform\ **Created:** [October 7, 2025, 11:41am UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071 "2025-10-07T11:41:09Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![MRead](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mread/32/2519_2.png) [@MRead](https://www.rubin.community/u/MRead)\ **Post date:** [October 7, 2025, 11:41am UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/1 "2025-10-07T11:41:09Z")
Hi We’ve encountered a minor issue with saving TAP results containing strings/Objects as FITS files. Attached is a simple notebook that replicates the error and the code is also pasted below. Basically just doing a small area search of obsCore, converting from pyVO results to astropy table and attempting to save as FITS. ```auto import numpy as np from lsst.rsp import get_tap_service service = get_tap_service("tap") assert service is not None query="SELECT * FROM ivoa.ObsCore WHERE CONTAINS(POINT('ICRS', 59.42914386, -48.90777109), s_region)=1" results = service.run_async(query) table = results.to_table() # Show headers print(table.colnames) table.info() table.write("testResults.fits", format="fits", overwrite=True) ``` The error is below, where is seems to not know the length of the string/Object columns and fails. I can get round the problem by changing all the Object type astropy colunms to fixed length strings. However wondering if anything has changed in one of the layers as we think this used to work? Or whether there is a better more robust way of going from TAP results to FITS file? Or if there is a better way of published the data (felis yaml file) that allows for the string length to be passed through? Thanks Mike ```auto --------------------------------------------------------------------------- VerifyError Traceback (most recent call last) Cell In[26], line 1 ----> 1 table.write("testResults.fits", format="fits", overwrite=True) File /opt/lsst/software/stack/conda/envs/lsst-scipipe-10.1.0/lib/python3.12/site-packages/astropy/table/connect.py:130, in TableWrite. __call__ (self, serialize_method, *args, **kwargs) 128 instance = self._instance 129 with serialize_method_as(instance, serialize_method): --> 130 self.registry.write(instance, *args, **kwargs) File /opt/lsst/software/stack/conda/envs/lsst-scipipe-10.1.0/lib/python3.12/site-packages/astropy/io/registry/core.py:386, in UnifiedOutputRegistry.write(self, data, format, *args, **kwargs) 381 format = self._get_valid_format( 382 "write", data. __class__ , path, fileobj, args, kwargs 383 ) 385 writer = self.get_writer(format, data. __class__ ) --> 386 return writer(data, *args, **kwargs) File /opt/lsst/software/stack/conda/envs/lsst-scipipe-10.1.0/lib/python3.12/site-packages/astropy/io/fits/connect.py:457, in write_table_fits(input, output, overwrite, append) 454 # Encode any mixin columns into standard Columns. 455 input = _encode_mixins(input) --> 457 table_hdu = table_to_hdu(input, character_as_bytes=True) 459 # Check if output file already exists 460 if isinstance(output, (str, os.PathLike)) and os.path.exists(output): File /opt/lsst/software/stack/conda/envs/lsst-scipipe-10.1.0/lib/python3.12/site-packages/astropy/io/fits/convenience.py:544, in table_to_hdu(table, character_as_bytes) 540 tarray.fill_value[colname] = "" 542 # TODO: it might be better to construct the FITS table directly from 543 # the Table columns, rather than go via a structured array. --> 544 table_hdu = BinTableHDU.from_columns( 545 tarray.filled(), header=hdr, character_as_bytes=character_as_bytes 546 ) 547 for col in table_hdu.columns: 548 # Binary FITS tables support TNULL *only* for integer data columns 549 # TODO: Determine a schema for handling non-integer masked columns 550 # with non-default fill values in FITS (if at all possible). 551 # Be careful that we do not set null for columns that were not masked! 552 int_formats = ("B", "I", "J", "K") File /opt/lsst/software/stack/conda/envs/lsst-scipipe-10.1.0/lib/python3.12/site-packages/astropy/io/fits/hdu/table.py:145, in _TableLikeHDU.from_columns(cls, columns, header, nrows, fill, character_as_bytes, **kwargs) 86 @classmethod 87 def from_columns( 88 cls, (...) 94 **kwargs, 95 ): 96 """ 97 Given either a `ColDefs` object, a sequence of `Column` objects, 98 or another table HDU or table data (a `FITS_rec` or multi-field (...) 143 `` __init__ `` may also be passed in as keyword arguments. 144 """ --> 145 coldefs = cls._columns_type(columns) 146 data = FITS_rec.from_columns( 147 coldefs, nrows=nrows, fill=fill, character_as_bytes=character_as_bytes 148 ) 149 hdu = cls( 150 data=data, header=header, character_as_bytes=character_as_bytes, **kwargs 151 ) File /opt/lsst/software/stack/conda/envs/lsst-scipipe-10.1.0/lib/python3.12/site-packages/astropy/io/fits/column.py:1495, in ColDefs. __init__ (self, input, ascii) 1492 self._init_from_coldefs(input._coldefs) 1493 elif isinstance(input, np.ndarray) and input.dtype.fields is not None: 1494 # Construct columns from the fields of a record array -> 1495 self._init_from_array(input) 1496 elif np.iterable(input): 1497 # if the input is a list of Columns 1498 self._init_from_sequence(input) File /opt/lsst/software/stack/conda/envs/lsst-scipipe-10.1.0/lib/python3.12/site-packages/astropy/io/fits/column.py:1572, in ColDefs._init_from_array(self, array) 1569 elif "K" in format: 1570 bzero = np.uint64(2**63) -> 1572 c = Column( 1573 name=cname, 1574 format=format, 1575 array=array.view(np.ndarray)[cname], 1576 bzero=bzero, 1577 dim=dim, 1578 ) 1579 self.columns.append(c) File /opt/lsst/software/stack/conda/envs/lsst-scipipe-10.1.0/lib/python3.12/site-packages/astropy/io/fits/column.py:675, in Column. __init__ (self, name, format, unit, null, bscale, bzero, disp, start, dim, array, ascii, coord_type, coord_unit, coord_ref_point, coord_ref_value, coord_inc, time_ref_pos) 672 for val in invalid_kwargs.values(): 673 msg.append(indent(val[1], 4 * " ")) --> 675 raise VerifyError("\n".join(msg)) 677 for attr in KEYWORD_ATTRIBUTES: 678 setattr(self, attr, valid_kwargs.get(attr)) VerifyError: The following keyword arguments to Column were invalid: Column format option (TFORMn) failed verification: Illegal format `P5A()`. The invalid value will be ignored for the purpose of formatting the data in this column. ``` @AndyJWil [TestStringsInTAP.ipynb](https://www.rubin.community/uploads/short-url/6gyocbL8NrhDpb73FBe1fS5wDue.ipynb) (21.8 KB) ---
**Author:** ![sfu](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/sfu/32/3060_2.png) [@sfu](https://www.rubin.community/u/sfu)\ **Post date:** [October 7, 2025, 6:49pm UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/2 "2025-10-07T18:49:00Z")
> [@MRead](#): > > `ivoa.ObsCore` Hi @MRead , thank you for the question. It looks like the issue comes from astropy. Perhaps it’s about how astropy deals with fits table columns (it seems the `P5A` format could not be processed). It could be a general question about TAP. Could you try other data structures for storing/saving the data? Maybe a pandas data frame, or saving it into a python pickle, or an hdf5 file. ---
**Author:** ![stvoutsin](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/stvoutsin/32/1218_2.png) [@stvoutsin](https://www.rubin.community/u/stvoutsin)\ **Post date:** [October 7, 2025, 11:13pm UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/3 "2025-10-07T23:13:58Z")
Hi @MRead ! Thanks for reporting this issue. Having briefly investigated it, my theory is that this may be a bug in astropy’s FITS writer when handling variable-length strings. The TAP service correctly returns strings with `arraysize="*"` in the VOTable, and `pyVO` converts these to a Python `object` dtype. However, when astropy tries to write these object dtype columns to FITS format, it attempts to use the P format (variable-length arrays) but generates an invalid format specification, `P5A()` instead of what I think should be `PA(max_length)` or maybe `max_lengthA`. I will bring this up with the astropy team either by raising an issue or proposing a PR once I do a bit more investigation into this just to confirm the theory. This may indeed have worked with previous version of pyVO if they handled variable-length strings differently, perhaps coercing to a different dtype. In the meantime I think your workaround of converting to fixed-length strings seems the most reasonable, but hopefully it is a temporary one if we can fix this upstream. Thanks again for catching it! ---
**Author:** ![MRead](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mread/32/2519_2.png) [@MRead](https://www.rubin.community/u/MRead)\ **Post date:** [October 8, 2025, 8:58am UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/4 "2025-10-08T08:58:51Z")
Hi @stvoutsin , thanks for looking into it. We think (not sure!) when it worked on our RSP (11 months ago), the strings in the FITS table came through with the correct lengths as declared in the SQL eg ```auto TTYPE7 = 'cc_flags' TFORM7 = '16A ' ``` in the yaml felis file this is/was described as ```auto - name: cc_flags "@id": "#matches_source.cc_flags" datatype: string description: cc_flags length: 16 mysql:datatype: VARCHAR(16) ``` is it possible that previously the TAP service retained the length=16 info i.e. `arraysize="16"` or some such rather than `arraysize="*"` Through the portal, when I save as `VOTable-FITS` it works and again an actual array size is written for the character columns ```auto Data product (file content) primary type Data product specific type ``` Anyhow we have a few work-arounds but it’d be good if these were not necessary 🙂 ---
**Author:** ![stvoutsin](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/stvoutsin/32/1218_2.png) [@stvoutsin](https://www.rubin.community/u/stvoutsin)\ **Post date:** [October 8, 2025, 7:10pm UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/5 "2025-10-08T19:10:22Z")
Okay thanks for the additional context. Assuming that column has a fixed length of 16, you could also add the `votable:arraysize` attribute to the yaml definition so that the result VOTable field definition has 16 instead of ‘\*’ as the arraysize: ```auto - name: cc_flags "@id": "#matches_source.cc_flags" datatype: string description: cc_flags length: 16 mysql:datatype: VARCHAR(16) votable:arraysize: 16 ``` While I haven’t testes this, it is possible that if this is a bug in astropy it may be specific to variable-length strings so this might just work. ---
**Author:** ![MRead](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mread/32/2519_2.png) [@MRead](https://www.rubin.community/u/MRead)\ **Post date:** [October 8, 2025, 8:46pm UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/6 "2025-10-08T20:46:00Z")
@stvoutsin Thanks, I’ll maybe give that a go if I can get things re-published. Assuming I’ve picked up the right yaml for obsCore `https://github.com/lsst/sdm_schemas/blob/main/python/lsst/sdm/schemas/ivoa_obscore.yaml` it does not use` votable:arraysize` just `length` and size info IS coming through to the VOTABLE-FITS output., though it must do some on the fly checking as the lengths are shorter? ```auto - name: dataproduct_type description: Data product (file content) primary type votable:utype: ObsDataset.dataProductType datatype: string length: 128 ivoa:ucd: meta.code.class tap:column_index: 10 tap:principal: 1 tap:std: 1 - name: dataproduct_subtype description: Data product specific type votable:utype: ObsDataset.dataProductSubtype datatype: string length: 64 ivoa:ucd: meta.code.class tap:column_index: 20 tap:principal: 1 tap:std: 1 ``` come through in teh VOTable-FITS as ```auto Data product (file content) primary type Data product specific type ``` ---
**Author:** ![stvoutsin](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/stvoutsin/32/1218_2.png) [@stvoutsin](https://www.rubin.community/u/stvoutsin)\ **Post date:** [October 9, 2025, 12:23am UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/7 "2025-10-09T00:23:25Z")
I’m assuming that there is something in the Firefly code that adds the fixed arraysizes for the VOTable (with FITS serialization) that doesn’t occur when downloading the tabledata VOTable from the portal, or when running a query through pyvo where you get `arraysize="*"` for those fields. I will ask internally and as well as the astropy team and report back. ---
**Author:** ![MRead](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mread/32/2519_2.png) [@MRead](https://www.rubin.community/u/MRead)\ **Post date:** [October 9, 2025, 5:55am UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/8 "2025-10-09T05:55:17Z")
Thanks @stvoutsin , just to be clear the issue we encountered is with saving to FITS file via notebook/astropy/pyvo. Saving to VOTable-FITS (VOTable header with FITS data?) via the portal works. (The portal doesn’t offer pure FITS). I presume pure VOTable works via the portal too. Cheers Mike ---
**Author:** ![MRead](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mread/32/2519_2.png) [@MRead](https://www.rubin.community/u/MRead)\ **Post date:** [October 24, 2025, 3:01pm UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/9 "2025-10-24T15:01:57Z")
> [@stvoutsin](#): > > `votable:arraysize: 16` Hi @svoutsinas , unfortunately adding this to the felis file for our database/catalog didn’t help with the FITS writing of the results. Note we also had to put the number in double quotes in the felis declaration as it complained > WARNING:felis.datamodel:Usage of an integer value for ‘votable:arraysize’ in column ‘cc\_flags’ is deprecated ---
**Author:** ![DouglasLTucker](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/douglasltucker/32/1341_2.png) [@DouglasLTucker](https://www.rubin.community/u/DouglasLTucker)\ **Post date:** [October 31, 2025, 10:12pm UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/10 "2025-10-31T22:12:40Z")
Hi @MRead ! It’s been a couple weeks since this topic has been active. We are glad you have some workarounds, but we just wanted to check if there is anything else we can do to help on our end here at Rubin. Let us know! Thanks! Best regards, Douglas ---
**Author:** ![MRead](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/mread/32/2519_2.png) [@MRead](https://www.rubin.community/u/MRead)\ **Post date:** [November 14, 2025, 9:38am UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/11 "2025-11-14T09:38:31Z")
HI @DouglasLTucker, that’s fine. Hopefully it’ll get fixed in due course, but no problem working round it. Cheers Mike ---
**Author:** ![MelissaGraham](https://sea2.discourse-cdn.com/flex002/user_avatar/www.rubin.community/melissagraham/32/2549_2.png) [@MelissaGraham](https://www.rubin.community/u/MelissaGraham)\ **Post date:** [December 18, 2025, 12:59am UTC](https://www.rubin.community/t/saving-strings-objects-in-tap-results-to-fits/11071/12 "2025-12-18T00:59:19Z")
Hi all, Forum admin here. Since suitable workarounds are in hand, I’ve marked @stvoutsin’s initial response as the solution for this particular topic. If these workarounds become insufficient, or fail in the future, please do open up a new topic so we can find a new solution (or new workaround).