SNOW-859689: DataFrame from flatten() function returns wrong schema()

[martin-frydl]

I have following code in Java:

        DataFrame orig = session.sql("SELECT 'a b c' AS SEQ");
        for (StructField f : orig.schema().fields()) {
            System.out.println("ORIG: " + f.name());
        }
        Column split = functions.callBuiltin("split", (Seq<Object>)(Seq<?>)JavaConverters.collectionAsScalaIterable(Arrays.asList(orig.col("SEQ"), functions.lit(" "))).toSeq());
        DataFrame flat = orig.flatten(split, "", true, false, "ARRAY");
        for (StructField f : flat.schema().fields()) {
            System.out.println("FLAT: " + f.name());
        }
        flat.explain();
        Row[] rows = flat.collect();

Here I create DataFrame from which I build another one using flatten() function. In original data frame there is a column SEQ which conflict with the same column generated by flatten. So Snowpark renames my column to some generated name. However, schema() called upon the flatten data frame does not return this name, it returns the original one. As a result, there are two SEQ columns returned. In previous versions of Snowpark like 1.6.2, this worked fine, schema was correct.

1.8.0 returns:

ORIG: SEQ <--- orig frame, just one column
FLAT: SEQ <---- SEQ as copy of original in flattened data frame
FLAT: SEQ <---- SEQ from flatten + other column from flatten
FLAT: KEY
FLAT: PATH
FLAT: INDEX
FLAT: VALUE
FLAT: THIS

The query is:

SELECT * FROM (
   -- Here SEQ is renamed
   SELECT  "SEQ" AS "a_4gKh_SEQ" FROM (
        SELECT 'a b c' AS SEQ
    )
), LATERAL FLATTEN (
   -- renamed column here
    INPUT => split("a_4gKh_SEQ", ' '),
    PATH => '', OUTER => true, RECURSIVE => false, MODE => 'ARRAY'
  )

In older Snowpark it printed something like this:

ORIG: SEQ <----------------- SEQ in orig frame
FLAT: "a_0uEn_SEQ" <---- renamed SEQ
FLAT: SEQ <------------------ SEQ from flatten
FLAT: KEY
FLAT: PATH
FLAT: INDEX
FLAT: VALUE
FLAT: THIS

[martin-frydl]

Seems like using output() instead of schema() gives correct names but I have no idea what is that function doing. I was not able to find any documentation and even the code (scala is quite a mystery for me, no 'def output' found by grep).

[sfc-gh-jfreeberg]

Possibly related to #42