基于SparkML2.0进行的Kaggle、JData等比赛.zip

##Elo Merchant Category Recommendation 该比赛是巴西的一个信用卡公司， ##源数据 ###train （201918条） - card_id 信用卡唯一标识符 - first_active_month 购买的记录时间，精确到月，格式为：'YYYY-MM' - feature_1 脱敏特征1 - feature_2 脱敏特征2 - feature_3 脱敏特征3 - target 在2个月之后的数据评分。Loyalty numerical score calculated 2 months after historical and evaluation period ... +------------------+---------------+---------+---------+---------+-----------+ |first_active_month|card_id |feature_1|feature_2|feature_3|target | +------------------+---------------+---------+---------+---------+-----------+ |2017-06 |C_ID_92a2005557|5 |2 |1 |-0.8202826 | |2017-01 |C_ID_3d0044924f|4 |1 |0 |0.39291325 | |2016-08 |C_ID_d639edf6cd|2 |2 |0 |0.68805599 | +------------------+---------------+---------+---------+---------+-----------+ ###test (123624条) 与train结构一致，没有target项 +------------------+---------------+---------+---------+---------+ |first_active_month|card_id |feature_1|feature_2|feature_3| +------------------+---------------+---------+---------+---------+ |2017-04 |C_ID_0ab67a22ab|3 |3 |1 | |2017-01 |C_ID_130fd0cbdd|2 |3 |0 | |2017-08 |C_ID_b709037bc5|5 |1 |1 | +------------------+---------------+---------+---------+---------+ ###historical_transaction（29112361，2千900万） 信用卡过去三个月的消费记录 - card_id - month_lag 距离参考日期的月份间隔 - purchase_date 购买时间 - authorized_flag 置信标签，Y是经过确认的，N是没有的，new中都是Y - installments 分期付款期数 - merchant_category_id 客户商品id (anonymized ) - subsector_id 商家商品种类id (anonymized ) - merchant_id 商家id（脱敏） - purchase_amount 正则化后的购买数量 - city_id 城市脱敏id - state_id 州脱敏id - category_1 脱敏商品1 - category_2 脱敏商品2 - category_3 脱敏商品3 ... +---------------+---------------+-------+----------+------------+----------+--------------------+---------------+---------+---------------+-------------------+----------+--------+------------+ |authorized_flag|card_id |city_id|category_1|installments|category_3|merchant_category_id|merchant_id |month_lag|purchase_amount|purchase_date |category_2|state_id|subsector_id| +---------------+---------------+-------+----------+------------+----------+--------------------+---------------+---------+---------------+-------------------+----------+--------+------------+ |Y |C_ID_4e6213e9bc|88 |N |0 |A |80 |M_ID_e020e9b302|-8 |-0.70333091 |2017-06-25 15:33:07|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |367 |M_ID_86ec983688|-7 |-0.73312848 |2017-07-15 12:10:45|1.0 |16 |16 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |80 |M_ID_979ed661fc|-6 |-0.720386 |2017-08-09 22:04:29|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |560 |M_ID_e6d5ae8ea6|-5 |-0.73535241 |2017-09-02 10:06:26|1.0 |16 |34 | +---------------+---------------+-------+----------+------------+----------+--------------------+---------------+---------+---------------+-------------------+----------+--------+------------+ ###new_merchant_period（196303，196万行） 信用卡当前月的消费记录 结构与historical一致 +---------------+---------------+-------+----------+------------+----------+--------------------+---------------+---------+---------------+-------------------+----------+--------+------------+ |authorized_flag|card_id |city_id|category_1|installments|category_3|merchant_category_id|merchant_id |month_lag|purchase_amount|purchase_date |category_2|state_id|subsector_id| +---------------+---------------+-------+----------+------------+----------+--------------------+---------------+---------+---------------+-------------------+----------+--------+------------+ |Y |C_ID_4e6213e9bc|88 |N |0 |A |80 |M_ID_e020e9b302|-8 |-0.70333091 |2017-06-25 15:33:07|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |367 |M_ID_86ec983688|-7 |-0.73312848 |2017-07-15 12:10:45|1.0 |16 |16 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |80 |M_ID_979ed661fc|-6 |-0.720386 |2017-08-09 22:04:29|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |560 |M_ID_e6d5ae8ea6|-5 |-0.73535241 |2017-09-02 10:06:26|1.0 |16 |34 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |80 |M_ID_e020e9b302|-11 |-0.72286538 |2017-03-10 01:14:19|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|333 |N |0 |A |80 |M_ID_50af771f8d|0 |-0.73488659 |2018-02-24 08:45:05|1.0 |9 |37 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |278 |M_ID_5e8220e564|-11 |-0.71685477 |2017-03-21 00:10:51|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|3 |N |0 |A |80 |M_ID_9d41786a50|-3 |-0.65704925 |2017-11-18 20:05:55|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |80 |M_ID_979ed661fc|-8 |-0.73796702 |2017-06-01 22:02:56|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |80 |M_ID_74ba14b5fc|-11 |-0.71535212 |2017-03-16 15:41:22|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |683 |M_ID_1449f22bfb|-9 |-0.73413526 |2017-05-09 12:42:07|1.0 |16 |34 | |Y |C_ID_4e6213e9bc|-1 |N |0 |A |560 |M_ID_7c5e93af2f|0 |-0.72792931 |2018-02-08 20:05:45|null |-1 |34 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |80 |M_ID_e020e9b302|-8 |-0.74164852 |2017-06-08 18:02:29|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |130 |M_ID_e8fb39882d|-7 |-0.72737333 |2017-07-14 12:59:38|1.0 |16 |41 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |80 |M_ID_97e86eae5f|-4 |-0.73188128 |2017-10-24 13:29:29|1.0 |16 |37 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |560 |M_ID_e6d5ae8ea6|-8 |-0.73819242 |2017-06-14 07:40:48|1.0 |16 |34 | |Y |C_ID_4e6213e9bc|69 |N |0 |A |879 |M_ID_00a6ca8a8a|0 |-0.72587068 |2018-02-07 12:19:33|1.0 |9 |29 | |Y |C_ID_4e6213e9bc|88 |N |0 |A |278 |M_ID_21e1552dab|-7 |-0.73488659 |201