Always access configuration through the appropriate session context rather than using global configuration access. This ensures that session-specific settings are respected and maintains consistency across different execution contexts.
Always access configuration through the appropriate session context rather than using global configuration access. This ensures that session-specific settings are respected and maintains consistency across different execution contexts.
Why this matters:
Preferred patterns:
// Pass SQLConf instance to constructors class JSONOptions(parameters: Map[String, String], sqlConf: SQLConf) { private val maxStringLen = sqlConf.getConf(SQLConf.JSON_MAX_STRING_LENGTH) }
2. **Access config through SparkSession:**
```scala
// Instead of using global SQLConf.get
val shuffleCleanupMode = determineShuffleCleanupMode(SQLConf.get)
// Use session-specific configuration
val shuffleCleanupMode = determineShuffleCleanupMode(sparkSession.sessionState.conf)
// Use session-specific timezone val zoneId = DateTimeUtils.getZoneId(SQLConf.get.sessionLocalTimeZone) ```
This pattern follows established practices in components like ParquetOptions
and ensures that configuration behavior is predictable and respects user-specified session settings.
Enter the URL of a public GitHub repository