Back to all reviewers

Session-specific configuration access

apache/spark
Based on 4 comments
Other

Always access configuration through the appropriate session context rather than using global configuration access. This ensures that session-specific settings are respected and maintains consistency across different execution contexts.

Configurations Other

Reviewer Prompt

Always access configuration through the appropriate session context rather than using global configuration access. This ensures that session-specific settings are respected and maintains consistency across different execution contexts.

Why this matters:

  • Different Spark sessions may have different configuration values
  • Global configuration access can ignore session-specific overrides
  • Proper session context ensures configuration consistency

Preferred patterns:

  1. Pass SQLConf instances explicitly: ```scala // Instead of accessing global config val maxStringLen = SQLConf.get.getConf(SQLConf.JSON_MAX_STRING_LENGTH)

// Pass SQLConf instance to constructors class JSONOptions(parameters: Map[String, String], sqlConf: SQLConf) { private val maxStringLen = sqlConf.getConf(SQLConf.JSON_MAX_STRING_LENGTH) }


2. **Access config through SparkSession:**
```scala
// Instead of using global SQLConf.get
val shuffleCleanupMode = determineShuffleCleanupMode(SQLConf.get)

// Use session-specific configuration
val shuffleCleanupMode = determineShuffleCleanupMode(sparkSession.sessionState.conf)
  1. Use session timezone settings: ```scala // Instead of hardcoded UTC or assumptions val zoneId = UTC

// Use session-specific timezone val zoneId = DateTimeUtils.getZoneId(SQLConf.get.sessionLocalTimeZone) ```

This pattern follows established practices in components like ParquetOptions and ensures that configuration behavior is predictable and respects user-specified session settings.

4
Comments Analyzed
Other
Primary Language
Configurations
Category

Source Discussions