<! Scala, a “maverick” language! –>

It’s hard to get started on Spark without touching Scala. Scala seems to provide Java with a lot of “functional-programming-like” syntactic sugar. Here are some of the things that make the language unique to you.

The reference materials mainly include:

  • Spark Big Data Analysis Technology (Scala Edition) [M]. Beijing University of Aeronautics and Astronautics Press, 2021. ISBN: 9787512433854
  • CHEN Huan, LIN Shifei. Spark Best Practices [M]. People’s Posts and Telecommunications Press, 2016

Scala’s basic ideas and considerations

SACLA Scalable Language is, as the name indicates, a Scalable programming Language:

  • Java-based virtual machine (Scala is compiled into JVM bytecode)
  • But it can be used both as a script and as a way to build large systems
  • Is a static language, but can support interactive programming just like a dynamic language
  • Faceted objects: Every value is an object, and every operation is a method call
  • Functional programming: All functions are objects, and functions are “first class citizens”
  • Almost everything in Scala is an expression

Scala is the interpreter and Scalac is the compiler; Scala Test. Scala Test. Scala & Scala Test. Scala Test. Scala Test. You can also enter Scala to enter the exchange programming interface.

So be aware that you need to install the JDK first and set the environment variable JAVA_HOME. Moreover, and more importantly, Scala is compatible with minor versions: 2.12.x and 2.13.x are not compatible, but 2.12.10 and 2.12.11 are.

The most basic syntax example

Type declarations, control structures (for, pattern matching, case)

Var one: Int = 1; var one: Int = 1; var one: String = '1'; Int, y: Int): Int = {x + y} var filename = if (! args.isEmpty) args(0) else "default.txt" for (i <- 1 to 4) println("iteration " + i)

1 to 4 is [1,2,3,4] and I until 4 is [1,2,3].

There are some other quirks about “for”.

// for (a < -1 to 2; B < - 1 to 2) {println (" a: "+ a +", b: "+ b)} / / results a: 1, b: 1 a: 1, b: 2 a: 2, b: 1 a: 2, b: If (x < -list1 if x % 2 == 1) print(" "+ x) // 3 5 1 7

When it comes to pattern matching, there are even more quirks. Here I refer directly to: case usage in Scala

// Set the value of the value to the value of the value.  val bools = List(true, false) for (bool <- bools) { bool match { case true => println("heads") case false => println("tails") case _ => println("something other than heads or tails (yikes!) ") } } import scala.util.Random val randomInt = new Random().nextInt(10) randomInt match { case 7 => println("lucky seven!" ) case otherNumber => println(" Boo, got boring ol' "+ otherNumber)} Val sundries = List(23, "Hello", 8.5, 'q') for (sundry < -sundry) {case I: Int => println("got an Integer: " + i) case s: String => println("got a String: " + s) case f: Double => println("got a Double: " + f) case other => println("got something else: Val willWork = List(1, 3, 23, 90) val willWork = List(4, 18, 18) 52) val empty = List() for (l <- List(willWork, willNotWork, empty)) { l match { case List(_, 3, _, _) => println("Four elements, With the 2nd being '3'.") case List(_*) => println("Any other List with 0 or more elements.")} Val tupA = ("Good", "Morning!") ) val tupB = ("Guten", "Tag!" ) for (tup <- List(tupA, tupB)) { tup match { case (thingOne, thingTwo) if thingOne == "Good" => println("A two-tuple starting with 'Good'.") case (thingOne, ThingTwo) =>println("This has two things: "+ thingOne +" and "+ thingTwo)}} String, age: Int) val alice = new Person("Alice", 25) val bob = new Person("Bob", 32) val charlie = new Person("Charlie", 32) for (person <- List(alice, bob, charlie)) { person match { case Person("Alice", 25) => println("Hi Alice!" ) case Person("Bob", 32) => println("Hi Bob!" ) case Person(name, age) => println("Who are you, " + age + " year-old person named " + name + "?" )}} // Six regular expressions matching val BookExtractorre = """Book: title=([^,]+),\s+authors=(.+)""".r val MagazineExtractorRE = """Magazine: title=([^,]+),\s+issue=(.+)""".r val catalog = List( "Book: title=Programming Scala, authors=Dean Wampler, Alex Payne", "Magazine: title=The New Yorker, issue=January 2009", "Book: title=War and Peace, authors=Leo Tolstoy", "Magazine: title=The Atlantic, issue=February 2009", "BadData: text=Who put this here??" ) for (item <- catalog) { item match { case BookExtractorRE(title, authors) => println("Book \"" + title + "\", written by " + authors) case MagazineExtractorRE(title, issue) => println("Magazine \"" + title + "\", issue " + issue) case entry => println("Unrecognized entry: " + entry) } }

With regard to case, I would like to emphasize its application in “unpacking” :

dict = Map("Piper" -> 95, "Bob" -> 90)
dict.foreach {
    case (k, v) => printf(
        "grade of %s is %s/n", k, v
    )
}

grade of Piper is 95
grade of Bob is 90

Foreach {case () => {}}. Note the braces for foreach. It’s equivalent to the following.

dict = Map("Piper" -> 95, "Bob" -> 90)
dict.foreach (
    x => println(
        s"grade of ${x._1} is ${x._2}"
    )
)

grade of Piper is 95
grade of Bob is 90

What’s unique about Scala’s syntax

  1. Parameterless methods are called without parentheses: args.isEmpty.

    Def width: Int = if (height == 0) 0 else contents(0).length width //
  2. forThe use of<-, which is equivalent to Pythonin 。
  3. Inheritance with keywordsextends ：class A(a: Int) extends B 。

  4. Singleton/static member variables and methods are defined in Object:

    object Timer { var count = 0 def currentCount() : LongTimer = {count += 1 count}}.currentCount(); }
  5. The function return does not have to be addedreturn, the default last expression.

  6. Functional: anonymous functions as arguments, and can be more concise

    val numbers = List(1, -3, -5, 9, Filter (x => x > 0) Numbers. Filter (_ > 0) // One argument that is only used once in the function

  7. _ has special meaning and work (placeholder)

    Adder (m: Int, n: Int) = m + n val add2 = adder(2, _: Int) // add2: (Int) => Int = < Function1 > add2(3) // Res1: Int = 5 Curriedsum (x: Int)(y: Int); Int) = x + y curriedSum(1) (2) var onePlus = curriedSum(1)_ case 1 => "one" case 2 => "two" case _ => "other" }

Scala’s Object Oriented and First-Class Citizen “Functions”

(1) + (2) / / 3

As above, (1) is an object and.+(2) is a method call. Everything in Scala is an object.

var increase = (x: Int) => x + 1

As above, functions are first-class citizens and can be assigned to variables.

Basic data structure

It has the following concepts:

  • Immutable listListAnd mutable listListBuffer
  • Fixed-length arrayArrayAnd variable-length arraysArrayBuffer
  • Immutable setSetAnd variable setscala.collection.mutable.Set
  • mappingMapAnd mutable mappingscala.collection.mutable.Map
  • tuplesTuple

Precautions and Scala’s oddities are tricky

  1. Using until is a good way to iterate through arrays. By and _* Special meaning:

    For (I < -0 until. Length) {} Array[Int](1 to 11 by 2:_*) // _*

  2. Use yield to generate the array

    val a = Array(1, 2, 3, 4)
val res1 = for (ele <- a) yield 2 * ele
// 2, 4, 6, 8

  3. The index of a tuple starts at 1

    val person = (1, 2, "ABC")
person._1  // 1

  4. Zipper operation zip

    val symbols = Array("<", "-", ">")
val counts = Array(2, 10, 2)
val pairs = symbols.zip(counts)
// Array[(String, Int)] = Array((<, 2), (-, 10), (>, 2))
for ((s, n) <- pairs) print(s * n)
<<---------->>
  5. MapMagic operation
/ / create the val dict = Map (95, "Piper" - > "Bob" - > 90) val kv = Map ((" Piper ", 95), (" Bob ", Dict ("Piper") // Combine ++ dict ++ kV dict.++(kV) // Add +, Remove -val n = dict +("Tom" -> 91) val l = dict - "Tom"

For mutable Map:

// += -= dict += (("Tom", 91), ("Jerry", 87)) dict -= "Tom" dict -= ("Jerry", "Bob") / / + + = - = associated with other collections dict + + = List ((" Tom ", 91), (87), "Jerry,") dict - = List (" Jerry ", "Bob")

  1. :: and ::: create lists

    1::3::5::Nil  // List[Int] = List(1, 3, 5)

Note :: is right-bound :(1::(3::(5::Nil))).

/ / : : : used to connect a List val L4 = L3: : : List (" Hadoop, Hbase ")

A discussion of data structures (List or Array?)

  • Use lists instead of arrays
  • The structure of lists is recursive (that is, linked lists,linkedList), while arrays are equal

Reference:

  • List, Array, ListBuffer, ArrayList, Set, and Tuple in Scala
  • Scala Learning Notes 5 (Collections)